Projet

Général

Profil

Mentiodns » Historique » Version 48

sacha, 20/10/2018 18:26

1 41 sacha
# Mentiodns
2 1 sacha
3
Validation des DNS à partir d'une liste sur un unbound et un DNS grand FAI et comparaison des résultats ;)
4
5 41 sacha
## Noeud actifs
6 1 sacha
7 41 sacha
| Nom       | Bloc                            |
8
| --------- | ------------------------------- |
9
| dam       | domain\_names.com\_sortedaf     |
10
| dam       | domain\_names.com\_sortedag     |
11
| dam       | domain\_names.com\_sortedah     |
12
| dam       | domain\_names.com\_sortedai     |
13
| mezzanine | domain\_names.com\_sortedad     |
14
| mezzanine | domain\_names.com\_sortedao     |
15
| mezzanine | domain\_names.com\_sortedap     |
16
| mezzanine | domain\_names.com\_sortedaq     |
17
| 1000i100  | domain\_names.com\_sortedab     |
18
| 1000i100  | domain\_names.com\_sortedaj     |
19
| 1000i100  | domain\_names.com\_sortedak     |
20
| 1000i100  | domain\_names.com\_sortedal     |
21
| sacha     | ~~domain\_names.org\_sortedaa~~ |
22
| sacha     | ~~domain\_names.org\_sortedab~~ |
23 48 sacha
| sacha     | domain\_names.org\_sortedac     |
24 41 sacha
| sacha     | ~~domain\_names.org\_sortedad~~ |
25 47 sacha
| sacha     | ~~domain\_names.net\_sortedac~~ |
26 41 sacha
| sacha     | domain\_names.net\_sortedad     |
27
| sacha     | ~~domain\_names.net\_sortedae~~ |
28 48 sacha
| sacha     | ~~domain\_names.com\_sortedac~~ |
29 44 sacha
| sacha     | domain\_names.com\_sortedae     |
30 45 sacha
| sacha     | domain\_names.com\_sortedam     |
31 41 sacha
| tazi      | domain\_names.com\_sortedan     |
32 47 sacha
| l         | domain\_names.com\_sortedar     |
33
| l         | domain\_names.org\_sortedae     |
34
| l         | ~~domain\_names.org\_sortedaf~~     |
35
| l         | domain\_names.net\_sortedaa     |
36
| l         | domain\_names.net\_sortedab     |
37 18 sacha
38 41 sacha
## Traités
39 35 sacha
40 43 sacha
domain\_names.org\_sortedaa 2000000
41
domain\_names.org\_sortedab 2000000
42
domain\_names.org\_sortedac 2000000
43 41 sacha
domain\_names.org\_sortedad 2000000
44 42 sacha
domain\_names.net\_sortedae 1937733
45 35 sacha
46 41 sacha
## Mentio
47 4 sacha
48 41 sacha
### mentio-check6
49 27 sacha
50 41 sacha
Packages: curl dig python socat tmux unbound whois
51 19 sacha
52 41 sacha
``` 
53 4 sacha
#-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+-#
54
#  MENTIODNS : Check for lying DNS (France)  #
55
#--------------------------------------------#
56 28 sacha
#  Version 1.6.1 date bug
57 16 sacha
#  Version 1.6 - conf file                   # 
58 13 sacha
#  Version 1.5 - test Dig resolving          #
59
#  Version 1.4 - Socat SSL sending results   # 
60
#  Version 1.3 - tld optioN                  #
61 1 sacha
#  Version 1.2 - Round robin on DNS_ISP_LIST #
62 28 sacha
#                For each request            #
63 4 sacha
#  Version 1.1 - Allow resume on basename    #
64
#  Version 1.0 - Parallel process with DIG   #
65
#--------------------------------------------#
66 1 sacha
# (c) Sacha at Aquilenet.fr part of FFDN.org #
67
#--------------------------------------------#
68
69
# This shity script intend to bruteforce the ISP lying DNS Servers to identify which one
70
# is going on Ministry of Interior Blocking page and compare the IP result from your favorite DNS server
71 16 sacha
# Use this script with the following parameters 
72
# $1 MODE: client server local
73
# $2 File source: list of domain names whithout tld
74
# $3 tld: com, org, ...
75
# $4 count number (if none from zero or from count file based on file name)
76
77
78 1 sacha
# If you relanch the script it will check if it has a counter for the given file to resume
79
# Blacklisted sites in $BLACKLIST_LOG file
80 13 sacha
# Diff ip from a domain name are in $DIFF_LOG 
81 1 sacha
82 13 sacha
# 1st launch creating config file
83 1 sacha
84 14 sacha
# Copy generated certificates:
85
# FILENAME=mentio_ssl-server 
86 1 sacha
# openssl genrsa -out $FILENAME.key 1024
87 13 sacha
# openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
88 1 sacha
# cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
89 13 sacha
# FILENAME=mentio_ssl-client
90
# ...
91
92 16 sacha
##########################################################
93 1 sacha
HOMEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
94 13 sacha
MENTIOCONF="$HOMEDIR/mentio.conf"
95 17 sacha
# Number of parallel requests thruw dig
96
parallel=10
97 16 sacha
##########################################################
98 1 sacha
99 16 sacha
100
##########################################################
101 13 sacha
### CHECK CONFIG
102
if [ ! -f $MENTIOCONF ]; then
103
echo "=================================================================="
104
echo "MENTIODNS"
105
echo "------------------------------------------------------------------" 
106
echo "1st time configuring"
107
echo -n "IP UNBOUND ? "
108
read DNS_MY
109 1 sacha
110 13 sacha
echo 'HOMEDIR="'$HOMEDIR'"' > $MENTIOCONF
111
echo 'DNS_MY="'$DNS_MY'"' >> $MENTIOCONF
112 1 sacha
IP_PUB=`curl ifconfig.io`
113 13 sacha
ASN=`whois -h whois.cymru.com $IP_PUB |cut -d' ' -f1|sed -n "2p"`
114 1 sacha
DNS_LIST="$HOMEDIR/mentio-DNS_ISP_LIST"
115
DNS_ISP_LIST=`grep $ASN $DNS_LIST|cut -d' ' -f3-`
116
echo 'DNS_ISP_LIST="'$DNS_ISP_LIST'"' >> $MENTIOCONF
117
echo "------------------------------------------------------------------"
118
echo " CONFIGURATION FILE:"
119
echo " please check and relaunch"
120
echo "------------------------------------------------------------------"
121
cat $MENTIOCONF
122
echo "------------------------------------------------------------------"
123
exit 1
124
fi
125 16 sacha
##########################################################
126 1 sacha
127 16 sacha
128
##########################################################
129
### PARAMETERS to execute the script
130
# Mode Log export with socat "client" "server" "local"
131
132
MODE=$1
133
134
if [ $MODE == "server" ]; then
135
socat -v -u openssl-listen:65522,fork,reuseaddr,cert=mentio_ssl-server.pem,cafile=mentio_ssl-client.crt OPEN:$HOMEDIR/MENTIO-DNS_DIFF,creat,append
136
exit 1
137
fi
138
##########################################################
139
140
141
##########################################################
142
### Check if commandline parameters are less than 3
143
144 1 sacha
if [ $# -lt 3 ]; then
145
echo "=================================================================="
146
echo "MENTIODNS"
147
echo "------------------------------------------------------------------"
148 14 sacha
echo "Missing Parameter, please enter:"
149 13 sacha
echo
150 1 sacha
echo "mentio-check client|server|local filename tld (count number)"
151 13 sacha
echo
152
exit 1
153
fi
154 16 sacha
##########################################################
155 13 sacha
156 16 sacha
157
##########################################################
158
### Get parameters
159
160
# From config file
161 1 sacha
source $MENTIOCONF
162 13 sacha
163 16 sacha
# From command line
164 1 sacha
# $2 DNS source file name
165 13 sacha
DNS_SOURCE=$2
166 1 sacha
# $3 TLD name (com, org...)
167 13 sacha
tld=$3
168 1 sacha
# line counter from the dns source file, nothing for auto-resuming
169
COUNT=$4
170
171 16 sacha
##########################################################
172 13 sacha
### SOCAT
173 29 sacha
SERVER="SOMEIP:65522"
174 13 sacha
SENDSOCAT="socat stdio openssl-connect:$SERVER,verify=0,cert=$HOMEDIR/mentio_ssl-client.pem,cafile=$HOMEDIR/mentio_ssl-server.crt"
175 16 sacha
##########################################################
176 13 sacha
### COLORS 
177
RED='\e[31m'
178
GREEN='\e[32m'
179
YELLOW='\e[33m'
180 1 sacha
GRAY='\e[90m'
181
NC='\033[0m' # No Color
182 16 sacha
##########################################################
183
### Various variables
184 13 sacha
DNS_SOURCE_BASENAME=`basename $DNS_SOURCE`
185 1 sacha
DIFF_LOG="$HOMEDIR/DNS_DIFF"
186
BLACKLIST_LOG="$HOMEDIR/DNS_BLACKLISTED"
187 13 sacha
lines=`wc -l $DNS_SOURCE|awk -F " " '{print $1}'`
188 4 sacha
countfile="$HOMEDIR/DNS_Count-$DNS_SOURCE_BASENAME"
189 16 sacha
##########################################################
190
### Dig parameters
191 1 sacha
DIG_FAST="+nodnssec +short +timeout=1 +tries=2"
192
DIG_SLOW="+nodnssec +short +timeout=5 +tries=3 "
193 16 sacha
##########################################################
194 13 sacha
195 16 sacha
196
##########################################################
197
### Generate list for dig: round robin from dns list
198
### Like (@DNS-server domain) x parallel 
199 1 sacha
_check(){
200
i=0
201 13 sacha
url=""
202
while [ $i -lt $parallel ]
203 1 sacha
do
204
n=`expr $count + $i`
205
ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
206
url="$url @$ISP_DNS `awk -v n="${n}" 'NR==n {print;exit}' $DNS_SOURCE`.$tld"
207
i=`expr $i + 1`
208
done
209
}
210 16 sacha
##########################################################
211 1 sacha
212
213 16 sacha
##########################################################
214
### Counter: create one if not existing, use existing instead
215 1 sacha
if [ -z $COUNT ]; then
216 28 sacha
        if [ -f $countfile ]; then
217
        count=`cat $countfile`
218
        else
219
        count=0
220
        echo $count > $countfile
221
        fi
222 13 sacha
else count=$COUNT
223
echo $count > $countfile
224 1 sacha
fi
225 4 sacha
##########################################################
226 16 sacha
227 4 sacha
228 16 sacha
##########################################################
229
### MAIN LOOP
230
231
while [ "$count" != "$lines" ]; do
232 4 sacha
echo $count > $countfile
233
_check
234 28 sacha
dateus=`date +%Y%m%d-%H%M%S`
235 13 sacha
site="$url"
236
echo "-------------------------------------------------------------------------------"
237
echo "#$count $dateus SITE:$site"
238
if nomentio=`dig @$DNS_MY $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio" ]; then
239 28 sacha
        echo -e "$GRAY Unknown zone $site $NC" 
240 13 sacha
fi
241
if mentio=`dig $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio" ]; then
242 28 sacha
        echo -e "$GRAY Unknown zone $site $NC"
243 1 sacha
fi
244
245
if [ -n "$nomentio" ] && [ -n "$mentio" ]; then
246
    if [ "$nomentio" != "$mentio" ]; then
247
        for i in $site; do
248
                if nomentio1=`dig $DIG_FAST @$DNS_MY $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio1" ]; then
249
                        echo -e "$GRAY Unknown zone $i $NC"
250
                fi
251
                ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
252
                if mentio1=`dig $DIG_FAST @$ISP_DNS $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio1" ]; then
253
                        echo -e "$GRAY Unknown zone $i $NC"
254
                fi
255
                        if [ "$nomentio1" != "$mentio1" ]; then
256
                                if [[ $mentio1 == 90.85.* ]]; then
257
                                        if [ $MODE == "client" ]; then
258
                                                echo "!!! $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" | $SENDSOCAT
259
                                        fi
260
                                        echo -e "$RED !!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
261
                                        echo "!!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $BLACKLIST_LOG
262
                                else
263
                                                if [ $MODE == "client" ]; then
264
                                                        echo ">>> $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" | $SENDSOCAT
265
                                                fi                   
266
                                        echo -e "$YELLOW >>> SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
267
                                        echo ">>> $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $DIFF_LOG
268
                                fi
269
                        fi
270
        done
271
    else
272
    echo -e "$GREEN#$count  SITE:$site  $NC"
273
    fi
274
fi
275
276
count=`expr $count + $parallel`
277
278
done
279
##########################################################
280 27 sacha
281 1 sacha
282
283 41 sacha
```
284 1 sacha
285 41 sacha
### mentio-DNS\_ISP\_LIST
286 27 sacha
287 41 sacha
    Bouygues 5410 194.158.122.10 194.158.122.15
288
    Free 12322 212.27.40.240 212.27.40.241 212.27.40.244 212.27.40.245
289
    Orange 3215 80.10.246.1 80.10.246.2 80.10.246.3 80.10.246.5 80.10.246.7 80.10.246.129 80.10.246.130 80.10.246.132 80.10.246.134 80.10.246.136 81.253.149.1 81.253.149.2 81.253.149.6 81.253.149.9 81.253.149.10
290
    SFR 15557 109.0.66.10 109.0.66.20
291 27 sacha
292 41 sacha
### mentio-monitor
293 27 sacha
294 41 sacha
    hosts="mentio-HOSTS"
295
    hosts_tmp="/tmp/mentio-HOSTS.tmp"
296
    rm -rf $hosts
297
    
298
    
299
    GREEN='\e[32m'
300
    RED='\e[31m'
301
    NC='\033[0m' # No Color  
302
    
303
    while true; do
304
            timeout 300 tail -n 0 -f MENTIO-DNS_DIFF | cut -d ' ' -f3|sort|uniq> $hosts_tmp
305
            mentiohosts=`cat $hosts_tmp`
306
            updates=`echo $mentiohosts|xargs -n 1`
307
            for i in $updates; do
308
                    exists=`grep "$i" $hosts`
309
                    if [ -z "$exists" ]; then
310
                            echo "ADD $i"
311
                            echo "$i"  >> $hosts 
312
                            sort -o $hosts $hosts
313
                            name=`echo $i|cut -d "." -f1`
314
                    fi
315
            done
316
    
317
    dateus=`date +%Y%m%d-%H%M%S`
318
    diffs=`diff --side-by-side --suppress-common-lines $hosts $hosts_tmp`
319
    echo "======================================================================"
320
    if [ -z "$diffs" ]; then
321
     echo -e "$GREEN $dateus - ALL HOSTS UP: $NC"
322
     echo -e "$GREEN `cat $hosts |xargs |sort` $NC"  
323
    else
324
     echo -e "$RED $dateus - MISSING HOST: $NC"
325
     echo -e "$RED $diffs $NC"
326
    fi
327
    done
328 18 sacha
329 41 sacha
## Test (valide au 14/06/18)
330 18 sacha
331 41 sacha
dig +short shahamat1.com  
332
90.85.16.52
333 18 sacha
334 41 sacha
## Vigies de la neutralité
335 18 sacha
336 41 sacha
https://ooni.torproject.org  
337 18 sacha
https://respectmynet.eu
338
339 41 sacha
## Cadre légal
340 15 sacha
341 41 sacha
https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000801164\&fastPos;=2\&fastReqId;=606073666\&categorieLien;=cid\&oldAction;=rechTexte#LEGIARTI000029756525  
342
le décret https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=LEGITEXT000030315036\&dateTexte;=20150305\&categorieLien;=cid#LEGITEXT000030315036  
343
https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=JORFTEXT000030195477\&dateTexte;=20180619
344 15 sacha
345 41 sacha
## Unbound
346 15 sacha
347 41 sacha
    server:
348
     verbosity: 1
349
     interface: 127.0.0.1
350
     do-ip4: yes
351
     do-ip6: no
352
     do-udp: yes
353
     do-tcp: no
354
     access-control: 127.0.0.0/8 allow 
355
     access-control: 0.0.0.0/0 refuse
356
     logfile: /var/log/unbound
357
     hide-identity: yes
358
     hide-version: yes
359
     harden-glue: yes
360
     use-caps-for-id: yes
361
     do-not-query-localhost: yes
362 15 sacha
363 41 sacha
## Lancement
364 15 sacha
365 41 sacha
tmux new-session -s foo -d "bash mentio-check6 client domain\_names.com\_sortedac com" \\; split-window -h "bash mentio-check6 client domain\_names.com\_sortedae com" \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedam com"\\; selectp -t 0 \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedan com"
366 15 sacha
367 41 sacha
## Ansible divers
368 15 sacha
369
Copy file:
370
371 41 sacha
ansible mentio -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
372 15 sacha
373
Copy file single host:
374
375 41 sacha
ansible mentio ---limit dam -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
376 15 sacha
377 41 sacha
ansible-playbook /etc/ansible/playbooks/mentio.yml
378 1 sacha
379 41 sacha
``` 
380 1 sacha
---
381
382
- hosts: mentio 
383
  sudo: no
384
  tasks:
385
    - name: copyfiles 
386
      copy:
387
        src: "{{ item.src }}"
388
        dest: "{{ item.dest }}"
389
      with_items:
390
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-check6',dest: '~/MENTIODNS/' }
391
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-DNS_ISP_LIST',dest: '~/MENTIODNS/' }
392
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.crt',dest: '~/MENTIODNS/' }
393
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.key',dest: '~/MENTIODNS/' }
394
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.pem',dest: '~/MENTIODNS/' }
395
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-server.crt',dest: '~/MENTIODNS/' }
396
397 41 sacha
```