Révision 41 - Historique - Mentiodns - Aquilenet - L'atelier d'Aquilenet

Mentiodns » Historique » Révision 41

Révision 40 (sacha, 14/07/2018 18:21) → Révision 41/59 (sacha, 03/09/2018 12:12)

# h1. Mentiodns 

 Validation des DNS à partir d'une liste sur un unbound et un DNS grand FAI et comparaison des résultats ;) 

 ## h2. Noeud actifs 

 | |_. Nom         | |_. Bloc                              | 
 | --------- | ------------------------------- | 
 | dam         | domain\_names.com\_sortedaf       | domain_names.com_sortedaf| 
 | dam         | domain\_names.com\_sortedag       | domain_names.com_sortedag| 
 | dam         | domain\_names.com\_sortedah       | domain_names.com_sortedah| 
 | dam         | domain\_names.com\_sortedai       | domain_names.com_sortedai| 
 | mezzanine | domain\_names.com\_sortedad       domain_names.com_sortedad | 
 | mezzanine | domain\_names.com\_sortedao       domain_names.com_sortedao | 
 | mezzanine | domain\_names.com\_sortedap       domain_names.com_sortedap | 
 | mezzanine | domain\_names.com\_sortedaq       domain_names.com_sortedaq | 
 | 1000i100    | domain\_names.com\_sortedab       domain_names.com_sortedab | 
 | 1000i100    | domain\_names.com\_sortedaj       domain_names.com_sortedaj | 
 | 1000i100    | domain\_names.com\_sortedak       domain_names.com_sortedak | 
 | 1000i100    | domain\_names.com\_sortedal       domain_names.com_sortedal | 
 | sacha       | ~~domain\_names.org\_sortedaa~~ -domain_names.org_sortedaa- | 
 | sacha       | ~~domain\_names.org\_sortedab~~ -domain_names.org_sortedab- | 
 | sacha       | ~~domain\_names.org\_sortedac~~ -domain_names.org_sortedac- | 
 | sacha       | ~~domain\_names.org\_sortedad~~ -domain_names.org_sortedad- | 
 | sacha       | domain\_names.net\_sortedac       domain_names.net_sortedac | 
 | sacha       | domain\_names.net\_sortedad       domain_names.net_sortedad | 
 | sacha       | ~~domain\_names.net\_sortedae~~ domain_names.net_sortedae | 
 | tazi        | domain\_names.com\_sortedac       domain_names.com_sortedac | 
 | tazi        | domain\_names.com\_sortedae       domain_names.com_sortedae | 
 | tazi        | domain\_names.com\_sortedam       domain_names.com_sortedam | 
 | tazi        | domain\_names.com\_sortedan       domain_names.com_sortedan | 
 | louisl      | domain_names.com_sortedar | domain\_names.com\_sortedar       | 
 | louisl      | domain\_names.org\_sortedae       domain_names.org_sortedae    | 
 | louisl      | domain\_names.org\_sortedaf       domain_names.org_sortedaf | 
 | louisl      | domain\_names.net\_sortedaa       domain_names.net_sortedaa | 
 | louisl      | domain\_names.net\_sortedab       domain_names.net_sortedab | 

 ## h2. Traités 

 domain\_names.org\_sortedaa domain_names.org_sortedaa 2000000   
 domain\_names.org\_sortedab  
 domain_names.org_sortedab 2000000   
 domain\_names.org\_sortedac  
 domain_names.org_sortedac 2000000   
 domain\_names.org\_sortedad 
 domain_names.org_sortedad 2000000 

 ## h2. Mentio 

 ### h3. mentio-check6 

 Packages: curl dig python socat tmux unbound whois 

 ```  
  

 <pre> 
 #-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+-# 
 #    MENTIODNS : Check for lying DNS (France)    # 
 #--------------------------------------------# 
 #    Version 1.6.1 date bug 
 #    Version 1.6 - conf file                     #  
 #    Version 1.5 - test Dig resolving            # 
 #    Version 1.4 - Socat SSL sending results     #  
 #    Version 1.3 - tld optioN                    # 
 #    Version 1.2 - Round robin on DNS_ISP_LIST # 
 #                  For each request              # 
 #    Version 1.1 - Allow resume on basename      # 
 #    Version 1.0 - Parallel process with DIG     # 
 #--------------------------------------------# 
 # (c) Sacha at Aquilenet.fr part of FFDN.org # 
 #--------------------------------------------# 

 # This shity script intend to bruteforce the ISP lying DNS Servers to identify which one 
 # is going on Ministry of Interior Blocking page and compare the IP result from your favorite DNS server 
 # Use this script with the following parameters  
 # $1 MODE: client server local 
 # $2 File source: list of domain names whithout tld 
 # $3 tld: com, org, ... 
 # $4 count number (if none from zero or from count file based on file name) 


 # If you relanch the script it will check if it has a counter for the given file to resume 
 # Blacklisted sites in $BLACKLIST_LOG file 
 # Diff ip from a domain name are in $DIFF_LOG  

 # 1st launch creating config file 

 # Copy generated certificates: 
 # FILENAME=mentio_ssl-server  
 # openssl genrsa -out $FILENAME.key 1024 
 # openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt 
 # cat $FILENAME.key $FILENAME.crt >$FILENAME.pem 
 # FILENAME=mentio_ssl-client 
 # ... 

 ########################################################## 
 HOMEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) 
 MENTIOCONF="$HOMEDIR/mentio.conf" 
 # Number of parallel requests thruw dig 
 parallel=10 
 ########################################################## 


 ########################################################## 
 ### CHECK CONFIG 
 if [ ! -f $MENTIOCONF ]; then 
 echo "==================================================================" 
 echo "MENTIODNS" 
 echo "------------------------------------------------------------------"  
 echo "1st time configuring" 
 echo -n "IP UNBOUND ? " 
 read DNS_MY 

 echo 'HOMEDIR="'$HOMEDIR'"' > $MENTIOCONF 
 echo 'DNS_MY="'$DNS_MY'"' >> $MENTIOCONF 
 IP_PUB=`curl ifconfig.io` 
 ASN=`whois -h whois.cymru.com $IP_PUB |cut -d' ' -f1|sed -n "2p"` 
 DNS_LIST="$HOMEDIR/mentio-DNS_ISP_LIST" 
 DNS_ISP_LIST=`grep $ASN $DNS_LIST|cut -d' ' -f3-` 
 echo 'DNS_ISP_LIST="'$DNS_ISP_LIST'"' >> $MENTIOCONF 
 echo "------------------------------------------------------------------" 
 echo " CONFIGURATION FILE:" 
 echo " please check and relaunch" 
 echo "------------------------------------------------------------------" 
 cat $MENTIOCONF 
 echo "------------------------------------------------------------------" 
 exit 1 
 fi 
 ########################################################## 


 ########################################################## 
 ### PARAMETERS to execute the script 
 # Mode Log export with socat "client" "server" "local" 

 MODE=$1 

 if [ $MODE == "server" ]; then 
 socat -v -u openssl-listen:65522,fork,reuseaddr,cert=mentio_ssl-server.pem,cafile=mentio_ssl-client.crt OPEN:$HOMEDIR/MENTIO-DNS_DIFF,creat,append 
 exit 1 
 fi 
 ########################################################## 


 ########################################################## 
 ### Check if commandline parameters are less than 3 

 if [ $# -lt 3 ]; then 
 echo "==================================================================" 
 echo "MENTIODNS" 
 echo "------------------------------------------------------------------" 
 echo "Missing Parameter, please enter:" 
 echo 
 echo "mentio-check client|server|local filename tld (count number)" 
 echo 
 exit 1 
 fi 
 ########################################################## 


 ########################################################## 
 ### Get parameters 

 # From config file 
 source $MENTIOCONF 

 # From command line 
 # $2 DNS source file name 
 DNS_SOURCE=$2 
 # $3 TLD name (com, org...) 
 tld=$3 
 # line counter from the dns source file, nothing for auto-resuming 
 COUNT=$4 

 ########################################################## 
 ### SOCAT 
 SERVER="SOMEIP:65522" 
 SENDSOCAT="socat stdio openssl-connect:$SERVER,verify=0,cert=$HOMEDIR/mentio_ssl-client.pem,cafile=$HOMEDIR/mentio_ssl-server.crt" 
 ########################################################## 
 ### COLORS  
 RED='\e[31m' 
 GREEN='\e[32m' 
 YELLOW='\e[33m' 
 GRAY='\e[90m' 
 NC='\033[0m' # No Color 
 ########################################################## 
 ### Various variables 
 DNS_SOURCE_BASENAME=`basename $DNS_SOURCE` 
 DIFF_LOG="$HOMEDIR/DNS_DIFF" 
 BLACKLIST_LOG="$HOMEDIR/DNS_BLACKLISTED" 
 lines=`wc -l $DNS_SOURCE|awk -F " " '{print $1}'` 
 countfile="$HOMEDIR/DNS_Count-$DNS_SOURCE_BASENAME" 
 ########################################################## 
 ### Dig parameters 
 DIG_FAST="+nodnssec +short +timeout=1 +tries=2" 
 DIG_SLOW="+nodnssec +short +timeout=5 +tries=3 " 
 ########################################################## 


 ########################################################## 
 ### Generate list for dig: round robin from dns list 
 ### Like (@DNS-server domain) x parallel  
 _check(){ 
 i=0 
 url="" 
 while [ $i -lt $parallel ] 
 do 
 n=`expr $count + $i` 
 ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1` 
 url="$url @$ISP_DNS `awk -v n="${n}" 'NR==n {print;exit}' $DNS_SOURCE`.$tld" 
 i=`expr $i + 1` 
 done 
 } 
 ########################################################## 


 ########################################################## 
 ### Counter: create one if not existing, use existing instead 
 if [ -z $COUNT ]; then 
         if [ -f $countfile ]; then 
         count=`cat $countfile` 
         else 
         count=0 
         echo $count > $countfile 
         fi 
 else count=$COUNT 
 echo $count > $countfile 
 fi 
 ########################################################## 


 ########################################################## 
 ### MAIN LOOP 

 while [ "$count" != "$lines" ]; do 
 echo $count > $countfile 
 _check 
 dateus=`date +%Y%m%d-%H%M%S` 
 site="$url" 
 echo "-------------------------------------------------------------------------------" 
 echo "#$count $dateus SITE:$site" 
 if nomentio=`dig @$DNS_MY $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio" ]; then 
         echo -e "$GRAY Unknown zone $site $NC"  
 fi 
 if mentio=`dig $DIG_SLOW $site|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio" ]; then 
         echo -e "$GRAY Unknown zone $site $NC" 
 fi 

 if [ -n "$nomentio" ] && [ -n "$mentio" ]; then 
     if [ "$nomentio" != "$mentio" ]; then 
         for i in $site; do 
                 if nomentio1=`dig $DIG_FAST @$DNS_MY $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$nomentio1" ]; then 
                         echo -e "$GRAY Unknown zone $i $NC" 
                 fi 
                 ISP_DNS=`echo $DNS_ISP_LIST | xargs -n 1| sort -R | head -n 1` 
                 if mentio1=`dig $DIG_FAST @$ISP_DNS $i|sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4| tr '\r\n' ' '` && [ -z "$mentio1" ]; then 
                         echo -e "$GRAY Unknown zone $i $NC" 
                 fi 
                         if [ "$nomentio1" != "$mentio1" ]; then 
                                 if [[ $mentio1 == 90.85.* ]]; then 
                                         if [ $MODE == "client" ]; then 
                                                 echo "!!! $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" | $SENDSOCAT 
                                         fi 
                                         echo -e "$RED !!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" 
                                         echo "!!! $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $BLACKLIST_LOG 
                                 else 
                                                 if [ $MODE == "client" ]; then 
                                                         echo ">>> $dateus `hostname` SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" | $SENDSOCAT 
                                                 fi                    
                                         echo -e "$YELLOW >>> SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1 $NC" 
                                         echo ">>> $dateus SITE:$i ISPDNS:$ISP_DNS REAL:$nomentio1 MENTIO:$mentio1" >> $DIFF_LOG 
                                 fi 
                         fi 
         done 
     else 
     echo -e "$GREEN#$count    SITE:$site    $NC" 
     fi 
 fi 

 count=`expr $count + $parallel` 

 done 
 ########################################################## 



 ``` 




 </pre> 

 ### mentio-DNS\_ISP\_LIST 

     h3. mentio-DNS_ISP_LIST 

 <pre> 
 Bouygues 5410 194.158.122.10 194.158.122.15 
     
 Free 12322 212.27.40.240 212.27.40.241 212.27.40.244 212.27.40.245 
     
 Orange 3215 80.10.246.1 80.10.246.2 80.10.246.3 80.10.246.5 80.10.246.7 80.10.246.129 80.10.246.130 80.10.246.132 80.10.246.134 80.10.246.136 81.253.149.1 81.253.149.2 81.253.149.6 81.253.149.9 81.253.149.10 
     
 SFR 15557 109.0.66.10 109.0.66.20 
 </pre> 

 ### h3. mentio-monitor 

     

 <pre> 
 hosts="mentio-HOSTS" 
     
 hosts_tmp="/tmp/mentio-HOSTS.tmp" 
     
 rm -rf $hosts 
    
    
     


 GREEN='\e[32m' 
     
 RED='\e[31m' 
     
 NC='\033[0m' # No Color   
    
       

 while true; do 
             
         timeout 300 tail -n 0 -f MENTIO-DNS_DIFF | cut -d ' ' -f3|sort|uniq> $hosts_tmp 
             
         mentiohosts=`cat $hosts_tmp` 
             
         updates=`echo $mentiohosts|xargs -n 1` 
             
         for i in $updates; do 
                     
                 exists=`grep "$i" $hosts` 
                     
                 if [ -z "$exists" ]; then 
                             
                         echo "ADD $i" 
                             
                         echo "$i"    >> $hosts  
                              
                         sort -o $hosts $hosts 
                             
                         name=`echo $i|cut -d "." -f1` 
                     
                 fi 
             
         done 
    
     

 dateus=`date +%Y%m%d-%H%M%S` 
     
 diffs=`diff --side-by-side --suppress-common-lines $hosts $hosts_tmp` 
     
 echo "======================================================================" 
     
 if [ -z "$diffs" ]; then 
      
  echo -e "$GREEN $dateus - ALL HOSTS UP: $NC" 
      
  echo -e "$GREEN `cat $hosts |xargs |sort` $NC"   
       
 else 
      
  echo -e "$RED $dateus - MISSING HOST: $NC" 
      
  echo -e "$RED $diffs $NC" 
     
 fi 
     
 done 
 </pre> 

 ## h2. Test (valide au 14/06/18) 

 

  dig +short shahamat1.com   
 
  90.85.16.52 

 ## h2. Vigies de la neutralité 

 https://ooni.torproject.org   
 
 https://respectmynet.eu 

 ## h2. Cadre légal 

 https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000801164\&fastPos;=2\&fastReqId;=606073666\&categorieLien;=cid\&oldAction;=rechTexte#LEGIARTI000029756525   
 https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000801164&fastPos;=2&fastReqId;=606073666&categorieLien;=cid&oldAction;=rechTexte#LEGIARTI000029756525         
 le décret https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=LEGITEXT000030315036\&dateTexte;=20150305\&categorieLien;=cid#LEGITEXT000030315036 https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=LEGITEXT000030315036&dateTexte;=20150305&categorieLien;=cid#LEGITEXT000030315036   
 https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=JORFTEXT000030195477\&dateTexte;=20180619 https://www.legifrance.gouv.fr/affichTexte.do;jsessionid=FE6BFDED672BF1E2EFC5CA70705CF26E.tplgfr21s_3?cidTexte=JORFTEXT000030195477&dateTexte;=20180619 

 ## h2. Unbound 

     

 <pre> 
 server: 
      
  verbosity: 1 
      
  interface: 127.0.0.1 
      
  do-ip4: yes 
      
  do-ip6: no 
      
  do-udp: yes 
      
  do-tcp: no 
      
  access-control: 127.0.0.0/8 allow  
       
  access-control: 0.0.0.0/0 refuse 
      
  logfile: /var/log/unbound 
      
  hide-identity: yes 
      
  hide-version: yes 
      
  harden-glue: yes 
      
  use-caps-for-id: yes 
      
  do-not-query-localhost: yes 
 </pre> 

 ## h2. Lancement 

 

  tmux new-session -s foo -d "bash mentio-check6 client domain\_names.com\_sortedac domain_names.com_sortedac com" \\; \; split-window -h "bash mentio-check6 client domain\_names.com\_sortedae domain_names.com_sortedae com" \\; \; split-window -v "bash mentio-check6 client domain\_names.com\_sortedam com"\\; domain_names.com_sortedam com"\; selectp -t 0 \\; \; split-window -v "bash mentio-check6 client domain\_names.com\_sortedan domain_names.com_sortedan com" 

 ## h2. Ansible divers 

 Copy file: 

 

  ansible mentio -m copy -a "src=mentio-check6 dest=~/MENTIODNS/" 

 


 Copy file single host: 

 

  ansible mentio ---limit --limit dam -m copy -a "src=mentio-check6 dest=~/MENTIODNS/" 

 ansible-playbook /etc/ansible/playbooks/mentio.yml 

 ```  
  


 <pre> 
 --- 

 - hosts: mentio  
   sudo: no 
   tasks: 
     - name: copyfiles  
       copy: 
         src: "{{ item.src }}" 
         dest: "{{ item.dest }}" 
       with_items: 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-check6',dest: '~/MENTIODNS/' } 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio-DNS_ISP_LIST',dest: '~/MENTIODNS/' } 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.crt',dest: '~/MENTIODNS/' } 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.key',dest: '~/MENTIODNS/' } 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-client.pem',dest: '~/MENTIODNS/' } 
         - { src: '/home/sacha/0nmyway/00_Aquilenet/FFDN/mentiodns.fr/mentio_ssl-server.crt',dest: '~/MENTIODNS/' } 

 ``` 
 


 </pre>

Projet

Général

Profil

Aquilenet

Mentiodns » Historique » Révision 41