Projet

Général

Profil

Mentiodns » Historique » Version 54

sacha, 10/11/2018 17:57

1 41 sacha
# Mentiodns
2 1 sacha
3 54 sacha
reCréation de la liste de censure gérée par le ministère de l'intérieur et envoyée automatiquement aux principaux FAI Français.
4
Validation des DNS à partir d'une liste exhaustive par tld. On vérifie si les résolutions DNS envoient sur les dns menteurs du ministère.
5 1 sacha
6 54 sacha
7 41 sacha
## Noeud actifs
8 1 sacha
9 41 sacha
| Nom       | Bloc                            |
10
| --------- | ------------------------------- |
11
| mezzanine | domain\_names.com\_sortedad     |
12
| mezzanine | domain\_names.com\_sortedao     |
13
| mezzanine | domain\_names.com\_sortedap     |
14
| mezzanine | domain\_names.com\_sortedaq     |
15
| 1000i100  | domain\_names.com\_sortedab     |
16
| 1000i100  | domain\_names.com\_sortedaj     |
17
| 1000i100  | domain\_names.com\_sortedak     |
18
| 1000i100  | domain\_names.com\_sortedal     |
19 1 sacha
| sacha     | ~~domain\_names.org\_sortedaa~~ |
20 41 sacha
| sacha     | ~~domain\_names.org\_sortedab~~ |
21 54 sacha
| sacha     | ~~domain\_names.org\_sortedac~~ |
22 41 sacha
| sacha     | ~~domain\_names.org\_sortedad~~ |
23 47 sacha
| sacha     | ~~domain\_names.net\_sortedac~~ |
24 41 sacha
| sacha     | domain\_names.net\_sortedad     |
25
| sacha     | ~~domain\_names.net\_sortedae~~ |
26 48 sacha
| sacha     | ~~domain\_names.com\_sortedac~~ |
27 1 sacha
| sacha     | domain\_names.com\_sortedae     |
28
| sacha     | domain\_names.com\_sortedaf     |
29 51 sacha
| sacha     | domain\_names.com\_sortedag     |
30 45 sacha
| sacha     | domain\_names.com\_sortedam     |
31 41 sacha
| tazi      | domain\_names.com\_sortedan     |
32 47 sacha
| l         | domain\_names.com\_sortedar     |
33 52 sacha
| l         | ~~domain\_names.org\_sortedae~~ |
34 47 sacha
| l         | ~~domain\_names.org\_sortedaf~~     |
35 1 sacha
| l         | domain\_names.net\_sortedaa     |
36
| l         | domain\_names.net\_sortedab     |
37 53 sacha
| l         | domain\_names.com\_sortedah     |
38
| l         | domain\_names.com\_sortedai     |
39 18 sacha
40 41 sacha
## Traités
41 35 sacha
42 43 sacha
domain\_names.org\_sortedaa 2000000
43
domain\_names.org\_sortedab 2000000
44
domain\_names.org\_sortedac 2000000
45 41 sacha
domain\_names.org\_sortedad 2000000
46 42 sacha
domain\_names.net\_sortedae 1937733
47 35 sacha
48 41 sacha
## Mentio
49 4 sacha
50 41 sacha
### mentio-check6
51 27 sacha
52 41 sacha
Packages: curl dig python socat tmux unbound whois
53 19 sacha
54 41 sacha
``` 
55 4 sacha
#-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+-#
56
#  MENTIODNS : Check for lying DNS (France)  #
57
#--------------------------------------------#
58 28 sacha
#  Version 1.6.1 date bug
59 16 sacha
#  Version 1.6 - conf file                   # 
60 13 sacha
#  Version 1.5 - test Dig resolving          #
61
#  Version 1.4 - Socat SSL sending results   # 
62
#  Version 1.3 - tld optioN                  #
63 1 sacha
#  Version 1.2 - Round robin on DNS_ISP_LIST #
64 28 sacha
#                For each request            #
65 4 sacha
#  Version 1.1 - Allow resume on basename    #
66
#  Version 1.0 - Parallel process with DIG   #
67
#--------------------------------------------#
68 1 sacha
# (c) Sacha at Aquilenet.fr part of FFDN.org #
69
#--------------------------------------------#
70
71
# This shity script intend to bruteforce the ISP lying DNS Servers to identify which one
72
# is going on Ministry of Interior Blocking page and compare the IP result from your favorite DNS server
73 16 sacha
# Use this script with the following parameters 
74
# $1 MODE: client server local
75
# $2 File source: list of domain names whithout tld
76
# $3 tld: com, org, ...
77
# $4 count number (if none from zero or from count file based on file name)
78
79
80 1 sacha
# If you relanch the script it will check if it has a counter for the given file to resume
81
# Blacklisted sites in $BLACKLIST_LOG file
82 13 sacha
# Diff ip from a domain name are in $DIFF_LOG 
83 1 sacha
84 13 sacha
# 1st launch creating config file
85 1 sacha
86 14 sacha
# Copy generated certificates:
87
# FILENAME=mentio_ssl-server 
88 1 sacha
# openssl genrsa -out $FILENAME.key 1024
89 13 sacha
# openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
90 1 sacha
# cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
91 13 sacha
# FILENAME=mentio_ssl-client
92
# ...
93
94 16 sacha
##########################################################
95 1 sacha
HOMEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
96 13 sacha
MENTIOCONF="$HOMEDIR/mentio.conf"
97 17 sacha
# Number of parallel requests thruw dig
98
parallel=10
99 16 sacha
##########################################################
100 1 sacha
101 16 sacha
102
##########################################################
103 13 sacha
### CHECK CONFIG
104
if [ ! -f $MENTIOCONF ]; then
105
echo "=================================================================="
106
echo "MENTIODNS"
107
echo "------------------------------------------------------------------" 
108
echo "1st time configuring"
109
echo -n "IP UNBOUND ? "
110
read DNS_MY
111 1 sacha
112 13 sacha
echo 'HOMEDIR="'$HOMEDIR'"' > $MENTIOCONF
113
echo 'DNS_MY="'$DNS_MY'"' >> $MENTIOCONF
114 1 sacha
IP_PUB=`curl ifconfig.io`
115 13 sacha
ASN=`whois -h whois.cymru.com $IP_PUB |cut -d' ' -f1|sed -n "2p"`
116 1 sacha
DNS_LIST="$HOMEDIR/mentio-DNS_ISP_LIST"
117
DNS_ISP_LIST=`grep $ASN $DNS_LIST|cut -d' ' -f3-`
118
echo 'DNS_ISP_LIST="'$DNS_ISP_LIST'"' >> $MENTIOCONF
119
echo "------------------------------------------------------------------"
120
echo " CONFIGURATION FILE:"
121
echo " please check and relaunch"
122
echo "------------------------------------------------------------------"
123
cat $MENTIOCONF
124
echo "------------------------------------------------------------------"
125
exit 1
126
fi
127 16 sacha
##########################################################
128 1 sacha
129 16 sacha
130
##########################################################
131
### PARAMETERS to execute the script
132
# Mode Log export with socat "client" "server" "local"
133
134
MODE=$1
135
136
if [ $MODE == "server" ]; then
137
socat -v -u openssl-listen:65522,fork,reuseaddr,cert=mentio_ssl-server.pem,cafile=mentio_ssl-client.crt OPEN:$HOMEDIR/MENTIO-DNS_DIFF,creat,append
138
exit 1
139
fi
140
##########################################################
141
142
143
##########################################################
144
### Check if commandline parameters are less than 3
145
146 1 sacha
if [ $# -lt 3 ]; then
147
echo "=================================================================="
148
echo "MENTIODNS"
149
echo "------------------------------------------------------------------"
150 14 sacha
echo "Missing Parameter, please enter:"
151 13 sacha
echo
152 1 sacha
echo "mentio-check client|server|local filename tld (count number)"
153 13 sacha
echo
154
exit 1
155
fi
156 16 sacha
##########################################################
157 13 sacha
158 16 sacha
159
##########################################################
160
### Get parameters
161
162
# From config file
163 1 sacha
source $MENTIOCONF
164 13 sacha
165 16 sacha
# From command line
166 1 sacha
# $2 DNS source file name
167 13 sacha
DNS_SOURCE=$2
168 1 sacha
# $3 TLD name (com, org...)
169 13 sacha
tld=$3
170 1 sacha
# line counter from the dns source file, nothing for auto-resuming
171
COUNT=$4
172
173 16 sacha
##########################################################
174 13 sacha
### SOCAT
175 29 sacha
SERVER="SOMEIP:65522"
176 13 sacha
SENDSOCAT="socat stdio openssl-connect:$SERVER,verify=0,cert=$HOMEDIR/mentio_ssl-client.pem,cafile=$HOMEDIR/mentio_ssl-server.crt"
177 16 sacha
##########################################################
178 13 sacha
### COLORS 
179
RED='\e[31m'
180
GREEN='\e[32m'
181
YELLOW='\e[33m'
182 1 sacha
GRAY='\e[90m'
183
NC='\033[0m' # No Color
184 16 sacha
##########################################################
185
### Various variables
186 13 sacha
DNS_SOURCE_BASENAME=`basename $DNS_SOURCE`
187 1 sacha
DIFF_LOG="$HOMEDIR/DNS_DIFF"
188
BLACKLIST_LOG="$HOMEDIR/DNS_BLACKLISTED"
189 13 sacha
lines=`wc -l $DNS_SOURCE|awk -F " " '{print $1}'`
190 4 sacha
countfile="$HOMEDIR/DNS_Count-$DNS_SOURCE_BASENAME"
191 16 sacha
##########################################################
192
### Dig parameters
193 1 sacha
DIG_FAST="+nodnssec +short +timeout=1 +tries=2"
194
DIG_SLOW="+nodnssec +short +timeout=5 +tries=3 "
195 16 sacha
##########################################################
196 13 sacha
197 16 sacha
198
##########################################################
199
### Generate list for dig: round robin from dns list
200
### Like (@DNS-server domain) x parallel 
201 1 sacha
_check(){
202
i=0
203 13 sacha
url=""
204
while [ $i -lt $parallel ]
205 1 sacha
do
206
n=`expr $count + $i`
207
ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
208
url="$url @$ISP_DNS `awk -v n="${n}" 'NR==n {print;exit}' $DNS_SOURCE`.$tld"
209
i=`expr $i + 1`
210
done
211
}
212 16 sacha
##########################################################
213 1 sacha
214
215 16 sacha
##########################################################
216
### Counter: create one if not existing, use existing instead
217 1 sacha
if [ -z $COUNT ]; then
218 28 sacha
        if [ -f $countfile ]; then
219
        count=`cat $countfile`
220
        else
221
        count=0
222
        echo $count > $countfile
223
        fi
224 13 sacha
else count=$COUNT
225
echo $count > $countfile
226 1 sacha
fi
227 4 sacha
##########################################################
228 16 sacha
229 4 sacha
230 16 sacha
##########################################################
231
### MAIN LOOP
232
233
while [ "$count" != "$lines" ]; do
234 4 sacha
echo $count > $countfile
235
_check
236 28 sacha
dateus=`date +%Y%m%d-%H%M%S`
237 13 sacha
site="$url"
238
echo "-------------------------------------------------------------------------------"
239
echo "#$count $dateus SITE:$site"
240
if nomentio=`dig @$DNS_MY $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio" ]; then
241 28 sacha
        echo -e "$GRAY Unknown zone $site $NC" 
242 13 sacha
fi
243
if mentio=`dig $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio" ]; then
244 28 sacha
        echo -e "$GRAY Unknown zone $site $NC"
245 1 sacha
fi
246
247
if [ -n "$nomentio" ] && [ -n "$mentio" ]; then
248
    if [ "$nomentio" != "$mentio" ]; then
249
        for i in $site; do
250
                if nomentio1=`dig $DIG_FAST @$DNS_MY $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio1" ]; then
251
                        echo -e "$GRAY Unknown zone $i $NC"
252
                fi
253
                ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
254
                if mentio1=`dig $DIG_FAST @$ISP_DNS $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio1" ]; then
255
                        echo -e "$GRAY Unknown zone $i $NC"
256
                fi
257
                        if [ "$nomentio1" != "$mentio1" ]; then
258
                                if [[ $mentio1 == 90.85.* ]]; then
259
                                        if [ $MODE == "client" ]; then
260
                                                echo "!!! $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" | $SENDSOCAT
261
                                        fi
262
                                        echo -e "$RED !!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
263
                                        echo "!!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $BLACKLIST_LOG
264
                                else
265
                                                if [ $MODE == "client" ]; then
266
                                                        echo ">>> $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" | $SENDSOCAT
267
                                                fi                   
268
                                        echo -e "$YELLOW >>> SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
269
                                        echo ">>> $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $DIFF_LOG
270
                                fi
271
                        fi
272
        done
273
    else
274
    echo -e "$GREEN#$count  SITE:$site  $NC"
275
    fi
276
fi
277
278
count=`expr $count + $parallel`
279
280
done
281
##########################################################
282 27 sacha
283 1 sacha
284
285 41 sacha
```
286 1 sacha
287 41 sacha
### mentio-DNS\_ISP\_LIST
288 27 sacha
289 41 sacha
    Bouygues 5410 194.158.122.10 194.158.122.15
290
    Free 12322 212.27.40.240 212.27.40.241 212.27.40.244 212.27.40.245
291
    Orange 3215 80.10.246.1 80.10.246.2 80.10.246.3 80.10.246.5 80.10.246.7 80.10.246.129 80.10.246.130 80.10.246.132 80.10.246.134 80.10.246.136 81.253.149.1 81.253.149.2 81.253.149.6 81.253.149.9 81.253.149.10
292
    SFR 15557 109.0.66.10 109.0.66.20
293 27 sacha
294 41 sacha
### mentio-monitor
295 27 sacha
296 41 sacha
    hosts="mentio-HOSTS"
297
    hosts_tmp="/tmp/mentio-HOSTS.tmp"
298
    rm -rf $hosts
299
    
300
    
301
    GREEN='\e[32m'
302
    RED='\e[31m'
303
    NC='\033[0m' # No Color  
304
    
305
    while true; do
306
            timeout 300 tail -n 0 -f MENTIO-DNS_DIFF | cut -d ' ' -f3|sort|uniq> $hosts_tmp
307
            mentiohosts=`cat $hosts_tmp`
308
            updates=`echo $mentiohosts|xargs -n 1`
309
            for i in $updates; do
310
                    exists=`grep "$i" $hosts`
311
                    if [ -z "$exists" ]; then
312
                            echo "ADD $i"
313
                            echo "$i"  >> $hosts 
314
                            sort -o $hosts $hosts
315
                            name=`echo $i|cut -d "." -f1`
316
                    fi
317
            done
318
    
319
    dateus=`date +%Y%m%d-%H%M%S`
320
    diffs=`diff --side-by-side --suppress-common-lines $hosts $hosts_tmp`
321
    echo "======================================================================"
322
    if [ -z "$diffs" ]; then
323
     echo -e "$GREEN $dateus - ALL HOSTS UP: $NC"
324
     echo -e "$GREEN `cat $hosts |xargs |sort` $NC"  
325
    else
326
     echo -e "$RED $dateus - MISSING HOST: $NC"
327
     echo -e "$RED $diffs $NC"
328
    fi
329
    done
330 18 sacha
331 41 sacha
## Test (valide au 14/06/18)
332 18 sacha
333 41 sacha
dig +short shahamat1.com  
334
90.85.16.52
335 18 sacha
336 41 sacha
## Vigies de la neutralité
337 18 sacha
338 41 sacha
https://ooni.torproject.org  
339 18 sacha
https://respectmynet.eu
340
341 41 sacha
## Cadre légal
342 15 sacha
343 41 sacha
https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000801164\&fastPos;=2\&fastReqId;=606073666\&categorieLien;=cid\&oldAction;=rechTexte#LEGIARTI000029756525  
344
le décret https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=LEGITEXT000030315036\&dateTexte;=20150305\&categorieLien;=cid#LEGITEXT000030315036  
345
https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=JORFTEXT000030195477\&dateTexte;=20180619
346 15 sacha
347 41 sacha
## Unbound
348 15 sacha
349 41 sacha
    server:
350
     verbosity: 1
351
     interface: 127.0.0.1
352
     do-ip4: yes
353
     do-ip6: no
354
     do-udp: yes
355
     do-tcp: no
356
     access-control: 127.0.0.0/8 allow 
357
     access-control: 0.0.0.0/0 refuse
358
     logfile: /var/log/unbound
359
     hide-identity: yes
360
     hide-version: yes
361
     harden-glue: yes
362
     use-caps-for-id: yes
363
     do-not-query-localhost: yes
364 15 sacha
365 41 sacha
## Lancement
366 15 sacha
367 41 sacha
tmux new-session -s foo -d "bash mentio-check6 client domain\_names.com\_sortedac com" \\; split-window -h "bash mentio-check6 client domain\_names.com\_sortedae com" \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedam com"\\; selectp -t 0 \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedan com"
368 15 sacha
369 41 sacha
## Ansible divers
370 15 sacha
371
Copy file:
372
373 41 sacha
ansible mentio -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
374 15 sacha
375
Copy file single host:
376
377 41 sacha
ansible mentio ---limit dam -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
378 15 sacha
379 41 sacha
ansible-playbook /etc/ansible/playbooks/mentio.yml
380 1 sacha
381 41 sacha
``` 
382 1 sacha
---
383
384
- hosts: mentio 
385
  sudo: no
386
  tasks:
387
    - name: copyfiles 
388
      copy:
389
        src: "{{ item.src }}"
390
        dest: "{{ item.dest }}"
391
      with_items:
392
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-check6',dest: '~/MENTIODNS/' }
393
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-DNS_ISP_LIST',dest: '~/MENTIODNS/' }
394
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.crt',dest: '~/MENTIODNS/' }
395
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.key',dest: '~/MENTIODNS/' }
396
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.pem',dest: '~/MENTIODNS/' }
397
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-server.crt',dest: '~/MENTIODNS/' }
398
399 41 sacha
```