Projet

Général

Profil

Mentiodns » Historique » Version 41

sacha, 03/09/2018 12:12

1 41 sacha
# Mentiodns
2 1 sacha
3
Validation des DNS à partir d'une liste sur un unbound et un DNS grand FAI et comparaison des résultats ;)
4
5 41 sacha
## Noeud actifs
6 1 sacha
7 41 sacha
| Nom       | Bloc                            |
8
| --------- | ------------------------------- |
9
| dam       | domain\_names.com\_sortedaf     |
10
| dam       | domain\_names.com\_sortedag     |
11
| dam       | domain\_names.com\_sortedah     |
12
| dam       | domain\_names.com\_sortedai     |
13
| mezzanine | domain\_names.com\_sortedad     |
14
| mezzanine | domain\_names.com\_sortedao     |
15
| mezzanine | domain\_names.com\_sortedap     |
16
| mezzanine | domain\_names.com\_sortedaq     |
17
| 1000i100  | domain\_names.com\_sortedab     |
18
| 1000i100  | domain\_names.com\_sortedaj     |
19
| 1000i100  | domain\_names.com\_sortedak     |
20
| 1000i100  | domain\_names.com\_sortedal     |
21
| sacha     | ~~domain\_names.org\_sortedaa~~ |
22
| sacha     | ~~domain\_names.org\_sortedab~~ |
23
| sacha     | ~~domain\_names.org\_sortedac~~ |
24
| sacha     | ~~domain\_names.org\_sortedad~~ |
25
| sacha     | domain\_names.net\_sortedac     |
26
| sacha     | domain\_names.net\_sortedad     |
27
| sacha     | ~~domain\_names.net\_sortedae~~ |
28
| tazi      | domain\_names.com\_sortedac     |
29
| tazi      | domain\_names.com\_sortedae     |
30
| tazi      | domain\_names.com\_sortedam     |
31
| tazi      | domain\_names.com\_sortedan     |
32
| louisl    | domain\_names.com\_sortedar     |
33
| louisl    | domain\_names.org\_sortedae     |
34
| louisl    | domain\_names.org\_sortedaf     |
35
| louisl    | domain\_names.net\_sortedaa     |
36
| louisl    | domain\_names.net\_sortedab     |
37 18 sacha
38 41 sacha
## Traités
39 35 sacha
40 41 sacha
domain\_names.org\_sortedaa 2000000  
41
domain\_names.org\_sortedab 2000000  
42
domain\_names.org\_sortedac 2000000  
43
domain\_names.org\_sortedad 2000000
44 35 sacha
45 41 sacha
## Mentio
46 4 sacha
47 41 sacha
### mentio-check6
48 27 sacha
49 41 sacha
Packages: curl dig python socat tmux unbound whois
50 19 sacha
51 41 sacha
``` 
52 4 sacha
#-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+-#
53
#  MENTIODNS : Check for lying DNS (France)  #
54
#--------------------------------------------#
55 28 sacha
#  Version 1.6.1 date bug
56 16 sacha
#  Version 1.6 - conf file                   # 
57 13 sacha
#  Version 1.5 - test Dig resolving          #
58
#  Version 1.4 - Socat SSL sending results   # 
59
#  Version 1.3 - tld optioN                  #
60 1 sacha
#  Version 1.2 - Round robin on DNS_ISP_LIST #
61 28 sacha
#                For each request            #
62 4 sacha
#  Version 1.1 - Allow resume on basename    #
63
#  Version 1.0 - Parallel process with DIG   #
64
#--------------------------------------------#
65 1 sacha
# (c) Sacha at Aquilenet.fr part of FFDN.org #
66
#--------------------------------------------#
67
68
# This shity script intend to bruteforce the ISP lying DNS Servers to identify which one
69
# is going on Ministry of Interior Blocking page and compare the IP result from your favorite DNS server
70 16 sacha
# Use this script with the following parameters 
71
# $1 MODE: client server local
72
# $2 File source: list of domain names whithout tld
73
# $3 tld: com, org, ...
74
# $4 count number (if none from zero or from count file based on file name)
75
76
77 1 sacha
# If you relanch the script it will check if it has a counter for the given file to resume
78
# Blacklisted sites in $BLACKLIST_LOG file
79 13 sacha
# Diff ip from a domain name are in $DIFF_LOG 
80 1 sacha
81 13 sacha
# 1st launch creating config file
82 1 sacha
83 14 sacha
# Copy generated certificates:
84
# FILENAME=mentio_ssl-server 
85 1 sacha
# openssl genrsa -out $FILENAME.key 1024
86 13 sacha
# openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
87 1 sacha
# cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
88 13 sacha
# FILENAME=mentio_ssl-client
89
# ...
90
91 16 sacha
##########################################################
92 1 sacha
HOMEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
93 13 sacha
MENTIOCONF="$HOMEDIR/mentio.conf"
94 17 sacha
# Number of parallel requests thruw dig
95
parallel=10
96 16 sacha
##########################################################
97 1 sacha
98 16 sacha
99
##########################################################
100 13 sacha
### CHECK CONFIG
101
if [ ! -f $MENTIOCONF ]; then
102
echo "=================================================================="
103
echo "MENTIODNS"
104
echo "------------------------------------------------------------------" 
105
echo "1st time configuring"
106
echo -n "IP UNBOUND ? "
107
read DNS_MY
108 1 sacha
109 13 sacha
echo 'HOMEDIR="'$HOMEDIR'"' > $MENTIOCONF
110
echo 'DNS_MY="'$DNS_MY'"' >> $MENTIOCONF
111 1 sacha
IP_PUB=`curl ifconfig.io`
112 13 sacha
ASN=`whois -h whois.cymru.com $IP_PUB |cut -d' ' -f1|sed -n "2p"`
113 1 sacha
DNS_LIST="$HOMEDIR/mentio-DNS_ISP_LIST"
114
DNS_ISP_LIST=`grep $ASN $DNS_LIST|cut -d' ' -f3-`
115
echo 'DNS_ISP_LIST="'$DNS_ISP_LIST'"' >> $MENTIOCONF
116
echo "------------------------------------------------------------------"
117
echo " CONFIGURATION FILE:"
118
echo " please check and relaunch"
119
echo "------------------------------------------------------------------"
120
cat $MENTIOCONF
121
echo "------------------------------------------------------------------"
122
exit 1
123
fi
124 16 sacha
##########################################################
125 1 sacha
126 16 sacha
127
##########################################################
128
### PARAMETERS to execute the script
129
# Mode Log export with socat "client" "server" "local"
130
131
MODE=$1
132
133
if [ $MODE == "server" ]; then
134
socat -v -u openssl-listen:65522,fork,reuseaddr,cert=mentio_ssl-server.pem,cafile=mentio_ssl-client.crt OPEN:$HOMEDIR/MENTIO-DNS_DIFF,creat,append
135
exit 1
136
fi
137
##########################################################
138
139
140
##########################################################
141
### Check if commandline parameters are less than 3
142
143 1 sacha
if [ $# -lt 3 ]; then
144
echo "=================================================================="
145
echo "MENTIODNS"
146
echo "------------------------------------------------------------------"
147 14 sacha
echo "Missing Parameter, please enter:"
148 13 sacha
echo
149 1 sacha
echo "mentio-check client|server|local filename tld (count number)"
150 13 sacha
echo
151
exit 1
152
fi
153 16 sacha
##########################################################
154 13 sacha
155 16 sacha
156
##########################################################
157
### Get parameters
158
159
# From config file
160 1 sacha
source $MENTIOCONF
161 13 sacha
162 16 sacha
# From command line
163 1 sacha
# $2 DNS source file name
164 13 sacha
DNS_SOURCE=$2
165 1 sacha
# $3 TLD name (com, org...)
166 13 sacha
tld=$3
167 1 sacha
# line counter from the dns source file, nothing for auto-resuming
168
COUNT=$4
169
170 16 sacha
##########################################################
171 13 sacha
### SOCAT
172 29 sacha
SERVER="SOMEIP:65522"
173 13 sacha
SENDSOCAT="socat stdio openssl-connect:$SERVER,verify=0,cert=$HOMEDIR/mentio_ssl-client.pem,cafile=$HOMEDIR/mentio_ssl-server.crt"
174 16 sacha
##########################################################
175 13 sacha
### COLORS 
176
RED='\e[31m'
177
GREEN='\e[32m'
178
YELLOW='\e[33m'
179 1 sacha
GRAY='\e[90m'
180
NC='\033[0m' # No Color
181 16 sacha
##########################################################
182
### Various variables
183 13 sacha
DNS_SOURCE_BASENAME=`basename $DNS_SOURCE`
184 1 sacha
DIFF_LOG="$HOMEDIR/DNS_DIFF"
185
BLACKLIST_LOG="$HOMEDIR/DNS_BLACKLISTED"
186 13 sacha
lines=`wc -l $DNS_SOURCE|awk -F " " '{print $1}'`
187 4 sacha
countfile="$HOMEDIR/DNS_Count-$DNS_SOURCE_BASENAME"
188 16 sacha
##########################################################
189
### Dig parameters
190 1 sacha
DIG_FAST="+nodnssec +short +timeout=1 +tries=2"
191
DIG_SLOW="+nodnssec +short +timeout=5 +tries=3 "
192 16 sacha
##########################################################
193 13 sacha
194 16 sacha
195
##########################################################
196
### Generate list for dig: round robin from dns list
197
### Like (@DNS-server domain) x parallel 
198 1 sacha
_check(){
199
i=0
200 13 sacha
url=""
201
while [ $i -lt $parallel ]
202 1 sacha
do
203
n=`expr $count + $i`
204
ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
205
url="$url @$ISP_DNS `awk -v n="${n}" 'NR==n {print;exit}' $DNS_SOURCE`.$tld"
206
i=`expr $i + 1`
207
done
208
}
209 16 sacha
##########################################################
210 1 sacha
211
212 16 sacha
##########################################################
213
### Counter: create one if not existing, use existing instead
214 1 sacha
if [ -z $COUNT ]; then
215 28 sacha
        if [ -f $countfile ]; then
216
        count=`cat $countfile`
217
        else
218
        count=0
219
        echo $count > $countfile
220
        fi
221 13 sacha
else count=$COUNT
222
echo $count > $countfile
223 1 sacha
fi
224 4 sacha
##########################################################
225 16 sacha
226 4 sacha
227 16 sacha
##########################################################
228
### MAIN LOOP
229
230
while [ "$count" != "$lines" ]; do
231 4 sacha
echo $count > $countfile
232
_check
233 28 sacha
dateus=`date +%Y%m%d-%H%M%S`
234 13 sacha
site="$url"
235
echo "-------------------------------------------------------------------------------"
236
echo "#$count $dateus SITE:$site"
237
if nomentio=`dig @$DNS_MY $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio" ]; then
238 28 sacha
        echo -e "$GRAY Unknown zone $site $NC" 
239 13 sacha
fi
240
if mentio=`dig $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio" ]; then
241 28 sacha
        echo -e "$GRAY Unknown zone $site $NC"
242 1 sacha
fi
243
244
if [ -n "$nomentio" ] && [ -n "$mentio" ]; then
245
    if [ "$nomentio" != "$mentio" ]; then
246
        for i in $site; do
247
                if nomentio1=`dig $DIG_FAST @$DNS_MY $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio1" ]; then
248
                        echo -e "$GRAY Unknown zone $i $NC"
249
                fi
250
                ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1`
251
                if mentio1=`dig $DIG_FAST @$ISP_DNS $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio1" ]; then
252
                        echo -e "$GRAY Unknown zone $i $NC"
253
                fi
254
                        if [ "$nomentio1" != "$mentio1" ]; then
255
                                if [[ $mentio1 == 90.85.* ]]; then
256
                                        if [ $MODE == "client" ]; then
257
                                                echo "!!! $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" | $SENDSOCAT
258
                                        fi
259
                                        echo -e "$RED !!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
260
                                        echo "!!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $BLACKLIST_LOG
261
                                else
262
                                                if [ $MODE == "client" ]; then
263
                                                        echo ">>> $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" | $SENDSOCAT
264
                                                fi                   
265
                                        echo -e "$YELLOW >>> SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC"
266
                                        echo ">>> $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $DIFF_LOG
267
                                fi
268
                        fi
269
        done
270
    else
271
    echo -e "$GREEN#$count  SITE:$site  $NC"
272
    fi
273
fi
274
275
count=`expr $count + $parallel`
276
277
done
278
##########################################################
279 27 sacha
280 1 sacha
281
282 41 sacha
```
283 1 sacha
284 41 sacha
### mentio-DNS\_ISP\_LIST
285 27 sacha
286 41 sacha
    Bouygues 5410 194.158.122.10 194.158.122.15
287
    Free 12322 212.27.40.240 212.27.40.241 212.27.40.244 212.27.40.245
288
    Orange 3215 80.10.246.1 80.10.246.2 80.10.246.3 80.10.246.5 80.10.246.7 80.10.246.129 80.10.246.130 80.10.246.132 80.10.246.134 80.10.246.136 81.253.149.1 81.253.149.2 81.253.149.6 81.253.149.9 81.253.149.10
289
    SFR 15557 109.0.66.10 109.0.66.20
290 27 sacha
291 41 sacha
### mentio-monitor
292 27 sacha
293 41 sacha
    hosts="mentio-HOSTS"
294
    hosts_tmp="/tmp/mentio-HOSTS.tmp"
295
    rm -rf $hosts
296
    
297
    
298
    GREEN='\e[32m'
299
    RED='\e[31m'
300
    NC='\033[0m' # No Color  
301
    
302
    while true; do
303
            timeout 300 tail -n 0 -f MENTIO-DNS_DIFF | cut -d ' ' -f3|sort|uniq> $hosts_tmp
304
            mentiohosts=`cat $hosts_tmp`
305
            updates=`echo $mentiohosts|xargs -n 1`
306
            for i in $updates; do
307
                    exists=`grep "$i" $hosts`
308
                    if [ -z "$exists" ]; then
309
                            echo "ADD $i"
310
                            echo "$i"  >> $hosts 
311
                            sort -o $hosts $hosts
312
                            name=`echo $i|cut -d "." -f1`
313
                    fi
314
            done
315
    
316
    dateus=`date +%Y%m%d-%H%M%S`
317
    diffs=`diff --side-by-side --suppress-common-lines $hosts $hosts_tmp`
318
    echo "======================================================================"
319
    if [ -z "$diffs" ]; then
320
     echo -e "$GREEN $dateus - ALL HOSTS UP: $NC"
321
     echo -e "$GREEN `cat $hosts |xargs |sort` $NC"  
322
    else
323
     echo -e "$RED $dateus - MISSING HOST: $NC"
324
     echo -e "$RED $diffs $NC"
325
    fi
326
    done
327 18 sacha
328 41 sacha
## Test (valide au 14/06/18)
329 18 sacha
330 41 sacha
dig +short shahamat1.com  
331
90.85.16.52
332 18 sacha
333 41 sacha
## Vigies de la neutralité
334 18 sacha
335 41 sacha
https://ooni.torproject.org  
336 18 sacha
https://respectmynet.eu
337
338 41 sacha
## Cadre légal
339 15 sacha
340 41 sacha
https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000801164\&fastPos;=2\&fastReqId;=606073666\&categorieLien;=cid\&oldAction;=rechTexte#LEGIARTI000029756525  
341
le décret https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=LEGITEXT000030315036\&dateTexte;=20150305\&categorieLien;=cid#LEGITEXT000030315036  
342
https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=JORFTEXT000030195477\&dateTexte;=20180619
343 15 sacha
344 41 sacha
## Unbound
345 15 sacha
346 41 sacha
    server:
347
     verbosity: 1
348
     interface: 127.0.0.1
349
     do-ip4: yes
350
     do-ip6: no
351
     do-udp: yes
352
     do-tcp: no
353
     access-control: 127.0.0.0/8 allow 
354
     access-control: 0.0.0.0/0 refuse
355
     logfile: /var/log/unbound
356
     hide-identity: yes
357
     hide-version: yes
358
     harden-glue: yes
359
     use-caps-for-id: yes
360
     do-not-query-localhost: yes
361 15 sacha
362 41 sacha
## Lancement
363 15 sacha
364 41 sacha
tmux new-session -s foo -d "bash mentio-check6 client domain\_names.com\_sortedac com" \\; split-window -h "bash mentio-check6 client domain\_names.com\_sortedae com" \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedam com"\\; selectp -t 0 \\; split-window -v "bash mentio-check6 client domain\_names.com\_sortedan com"
365 15 sacha
366 41 sacha
## Ansible divers
367 15 sacha
368
Copy file:
369
370 41 sacha
ansible mentio -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
371 15 sacha
372
Copy file single host:
373
374 41 sacha
ansible mentio ---limit dam -m copy -a "src=mentio-check6 dest=~/MENTIODNS/"
375 15 sacha
376 41 sacha
ansible-playbook /etc/ansible/playbooks/mentio.yml
377 1 sacha
378 41 sacha
``` 
379 1 sacha
---
380
381
- hosts: mentio 
382
  sudo: no
383
  tasks:
384
    - name: copyfiles 
385
      copy:
386
        src: "{{ item.src }}"
387
        dest: "{{ item.dest }}"
388
      with_items:
389
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-check6',dest: '~/MENTIODNS/' }
390
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-DNS_ISP_LIST',dest: '~/MENTIODNS/' }
391
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.crt',dest: '~/MENTIODNS/' }
392
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.key',dest: '~/MENTIODNS/' }
393
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.pem',dest: '~/MENTIODNS/' }
394
        - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-server.crt',dest: '~/MENTIODNS/' }
395
396 41 sacha
```