Internet / WWW

Index

General

3D

Analítica / Analytics

Aplicacions / Applications

Banda ampla / Broadband

CMS (content management) 

Correu electrònic / E-mail 

Dominis / Domains

Allotjament / Hosting

apps
 manage
 url
 capacitat
 baixada
 comments
Google Drive
 -
https://googledrive.com/host/0B42MQgikjmMwT05OV09ONDR4WkE/

Is it possible to have clean URLs for Google Drive items? [closed]
gdriv.es
http://gdriv.es/francescpinyol/

only highest level url
gweb.io



$1.50/month
Dropbox -
https://dl.dropboxusercontent.com/u/8062953/site/index.html
 (Dropbox)

dockbox.io



dockbox.io – Host Websites in Dropbox With Custom Domains
harp.io



Harp.io – Host production websites with custom domains and SSL
KISSr
 http://www.kissr.com/sites
 http://francescpinyol.kissr.com/

 50MB
 50MB/mes
DropPages



based on given templates
Site44


100MB/mes
allotjament.cat





webgratis.cat

http://francescpinyol.webgratis.cat/
 1GB
 100GB/mes
 PHP, MySQL


alta domini
 renovació (¤/any)
emmagatzematge
 tràfic
 correu
 altres
 preus






comptes
 espai
 MySQL PHP
 One Click
 mensual
 anual
active24
 8
 16,99 8
 Basic Hosting
 5GB
 il·limitat

x
 x
 x
 4,99 1,99
swhosting
 8
 17,75
 Petit L
 2GB
 5GB/mes

-
 x
 -
 3,50
Don Dominio (.cat)
 9,95
 22,95 17,95
 Pla Mini
 100MB
 5GB/mes 5
 50MB
 -
 -
 -
 0,08
 1
Pla Basic
 1GB
 12GB/mes
 100
 300MB
 100MB
 x
2,08
 25
Dina Hosting
 8,95
 18,60
 Hosting Personal
 2GB
 30GB/mes
 20



4,50
Virtual Pyme
 8,95
 18,95
 Mínim
 1GB 5GB/mes 10



2,45
CD Mon
 8
 19,95
 Pla Start
 1GB
 10GB/mes
 1
-
 x
2,50
 30
Entorno Digital
 8
 19,99
 Pla Presència
 2GB
 10GB/mes 5
-
 -
3,80

Feeds

Geolocalització / Geolocation

Grups de notícies / Newsgroups

Gestors de preferits / Bookmarks managers

Internet per satèl·lit / Internet via satellite

Internet Services Providers (ISP)

Internet TV

Internet of things

P2P 

Protocols

  • TCP and UDP port numbers (Wikipedia) nou
  • Internet Protocol (IP)
  • W3C
    • HTTP
    • HTTPS
      • Securing the web
      • Servidors / Servers
      • Clients
      • Server Name Indicator (SNI) (wp)
        • from nginx docs:
          • "With this configuration a browser receives the default server’s certificate, i.e. www.example.com regardless of the requested server name. This is caused by SSL protocol behaviour. The SSL connection is established before the browser sends an HTTP request and nginx does not know the name of the requested server. Therefore, it may only offer the default server’s certificate."
          • "A more generic solution for running several HTTPS servers on a single IP address is TLS Server Name Indication extension (SNI, RFC 6066), which allows a browser to pass a requested server name during the SSL handshake and, therefore, the server will know which certificate it should use for the connection."
        • SNI in nginx
      • HSTS
      • Marca blanca / White label
      • Let's Encrypt
        • wildcards
        • Protocols
        • Documentation
          • Challenge types
            • ACME
              challenge              		 used by pros cons

              		certbot plugin

              HTTP-01
              • apache
              • nginx
              • webroot
              • standalone
              • manual
              • It allows hosting providers to issue certificates for domains CNAMEd to them.
              • ...
              DNS-01
              • DNS plugins
                • ...
              • manual





        • ACME Clients
          • certbot
            • Documentation
            • Installation
            • Compilation
              • Requirements
                • CentOS
                  • sudo yum install python2-mock python-zope-interface python-zope-component python-six python-setuptools python2-acme
                • Mageia
                  • Compile certbot (recommended)
                  • or get it:
                    • wget https://dl.eff.org/certbot-auto
                    • chmod a+x certbot-auto
                    • ./certbot-auto -debug
                    • ./certbot-auto certonly
              • Download
                • git clone https://github.com/certbot/certbot.git
              • Compile
                • cd certbot
                • sudo python setup.py install
              • Compile version >=0.22 (for wildcards)
                • cd certbot
                • virtualenv env
                • source env/bin/activate
                • pip install --upgrade pip
                • pip install --upgrade setuptools
                • cd acme
                • python setup.py install
                • cd ..
                • python setup.py install
                • sudo ./env/bin/certbot ...
            • Configuració / Setup
              • serveis amb usuari no root / non-root services
                • How to use certs in non-root services?
                • letsencrypt_change_permissions.sh
                  • #!/bin/bash
                    # file permissions
                    chgrp ssl-cert -R /etc/letsencrypt
                    chmod 2755 /etc/letsencrypt
                    chmod g=rX -R /etc/letsencrypt
                • groupadd --gid 3000 ssl-cert change_letsencrypt_persmissions.sh
                  usermod -a -G ssl-cert nginx
                • crontab
                  • certbot -n renew --deploy-hook /usr/local/bin/letsencrypt_change_permissions.sh
            • Usage

              		info
              		 usage from
              		 additional setup for running server


              		CLI options /etc/letsencrypt/renewal/your.domain.org.conf
              		 Apache Nginx


              		certbot ...
              		 [renewalparams]
              		 /etc/httpd/httpd.conf
              		 /etc/nginx/nginx.conf
              authentication
              		 when your html pages are on a static server (hosted, AWS S3...)
              		 --authenticator manual


              when you have no running server, but ports 80 and 443 are available. A mini server will be started just for the process.
              E.g.: Postfix email server
              		 --authenticator standalone authenticator = standalone

              when you have a running server (certbot will generate some content locally, in /path/to/.well-known/, which must be available by http from letsencrypt servers; this requires additional setup of your server)
              Can also be used to obtain a certificate for Postfix (-d mail.domain.org) when an http server is already running (even if it is  running with server_name your.domain.org). But make sure that you have a DNS entry: mail.domain.org CNAME your.domain.org               		--authenticator webroot -w /path/to
              		 authenticator = webroot
              [[webroot_map]]
              your.domain.org = /path/to
              		 (needed?)
              <VirtualHost *:80>
                  Alias /.well-known /path/to/.well-known

                  <Directory "/path/to/.well-known"> 
                      Options Indexes FollowSymLinks MultiViews 
                      AllowOverride All 
                      Require all granted   
                  </Directory>

              		 server {
                  listen      80;
                 
                  server_name         your.domain.org;

                  # letsencrypt
                  location /.well-known {
                      alias /path/to/.well-known;
                  }
              when you have a running Apache server --authenticator apache authenticator = apache

              when you have a running Nginx server --authenticator nginx authenticator = nginx







              installation
              		 you do not want to install the obtained certificate
              		 certonly
              		 installer = None

              you want to install the obtained certificate in the Apache config
              		 --installer apache
              		 installer = apache

              you want to install the obtained certificate in the Nginx config --installer nginx
              		 installer = nginx

              • Examples
                • Just retrieve a certificate (a nginx server is running):
                  • webroot=/usr/share/nginx/html
                    letsencrypt_port=80
                    key_size=4096
                    email=me@my_company.com
                    domain=my_subdomain.my_company.com
                    sudo certbot certonly -n --agree-tos --webroot -w $webroot --http-01-port $letsencrypt_port --rsa-key-size $key_size --email $email -d $domain
                • just retrieve a wildcard certificate (requires certbot version >=0.22)
                  • install the required dns plugin
                  • sudo -i
                    letsencrypt_port=80
                    key_size=4096
                    email=me@my_company.com
                    domain='*.my_company.com'
                    certbot certonly -n --agree-tos --dns-route53 --http-01-port $letsencrypt_port --rsa-key-size $key_size --email $email -d $domain                     --server https://acme-v02.api.letsencrypt.org/directory
                  • Problemes / Problems
                    • The currently selected ACME CA endpoint does not support issuing wildcard certificates.
                      • Solució / Solution
                        • --server https://acme-v02.api.letsencrypt.org/directory
                • just retrieve a certificate using AWS Route 53
                  • install the required dns plugin
                  • sudo -i
                    email=me@my_company.com
                    domain='mydomain.my_company.com'
                    certbot certonly [--dry-run] -n --agree-tos --dns-route53 --key-type ecdsa --email $email -d $domain
              • Renewal
                • Renewing certificates
                  • --pre-hook
                  • --post-hook
                  • --deploy-hook
                • all certificates:
                  • certbot -n renew
                • specified certificates:
                  • certbot certonly -n --authenticator webroot --webroot-path /path/to -d your.domain.org
          • Client options
        • Apache
          • first time
            • certbot run --apache --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • non interactive:
              • certbot run --apache -n --agree-tos --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
          • first renewal (because apache authenticator is not working for certain apache config)
            • certbot certonly -n --authenticator webroot --webroot-path /path/to/ -d your.domain.org
          • next renewals
            • sudo certbot -n renew
            • Problemes / Problems
              • /etc/letsencrypt/options-ssl-apache.conf not found
                • Solució / Solution
                  • sudo yum install python2-certbot-apache
                  • ln -s /usr/lib/python2.7/site-packages/certbot_apache/options-ssl-apache.conf /etc/letsencrypt/options-ssl-apache.conf
              • Could not open configuration file /etc/letsencrypt/options-ssl-apache.conf: Permission denied
                • This happens because /etc/letsencrypt is a symbolic link to /mnt/nfs/letsencrypt, a nfs-mounted point
                • Solució / Solution
        • Nginx (using webroot)
          • first time
            • certbot certonly --webroot -w /path/to --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • non interactive:
              • certbot certonly -n --agree-tos --webroot -w /path/to --http-01-port 80--rsa-key-size 4096 --email your@email.org -d your.domain.org
          • renewal
            • /etc/nginx/conf.d/my_site.conf
              • server {
                    # letsencrypt
                    location /.well-known {
                        alias /path/to/.well-known;
                    }
                    ...
                }
            • check that /etc/letsencrypt/renewal/my.site.org.conf contains:
              • ...
                [renewalparams]
                authenticator = webroot
                webroot_path = /path/to
                installer = None
                ...
            • sudo certbot renew --post-hook "systemctl restart nginx.service"
            • crontab
              • certbot -q -n renew --post-hook "systemctl restart nginx.service"
        • Certificate installation in hosting services
          1. Get only the certificate (follow the instructions)
            • domain=my_subdomain.my_company.org
              sudo certbot certonly --authenticator manual -d $domain
          2. Install it:
            • DonDominio
              • Alojamiento web / Subdominios
                • CERTIFICADO:
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/cert.pem
                  • CERTIFICADO: CTRL-V
                • CLAVE SSL: 
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/privkey.pem
                  • CLAVE SSL: CTRL-V
                • CERTIFICADOS DE AUTORIDAD: 
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/chain.pem
                  • CERTIFICADOS DE AUTORIDAD: CTRL-V
                  • elimineu el segon certificat que acabeu d'enganxar dins de la casella CERTIFICADOS DE AUTORIDAD
                  • 247
              • panel247
        • AWS
          • if you want to use certificates for LoadBalancer or CloudFront, consider using AWS ACM instead
          • EC2
          • Elastic Load Balancer (ELB)
            • Let's encrypt and ELBs?
            • First time
              1. create a load balancer with a port 80 listener
              2. AWS IAM: grant permissions to user/role that will be used in next steps
                • elasticloadbalancing:CreateLoadBalancerListeners
                • iam:GetServerCertificate
                • iam:UploadServerCertificate
                • Example of policy:
                  • {
                        "Version": "2012-10-17",
                        "Statement": [
                            {
                                "Sid": "",
                                "Effect": "Allow",
                                "Action": [
                                    "elasticloadbalancing:DescribeLoadBalancers",
                                    "elasticloadbalancing:CreateLoadBalancerListeners",
                                    "elasticloadbalancing:SetLoadBalancerListenerSSLCertificate"
                                ],
                                "Resource": [
                                    "*"
                                ]
                            },
                            {
                                "Sid": "",
                                "Effect": "Allow",
                                "Action": [
                                    "iam:ListServerCertificates",
                                    "iam:GetServerCertificate",
                                    "iam:UploadServerCertificate"
                                ],
                                "Resource": [
                                    "*"
                                ]
                            }
                        ]
                    }
              3. from an instance (e.g. from an instance behind the load balancer)
                1. get a LE certificate (IMPORTANT: keys with 4096 bytes are not allowed; only 2048 or 1024 (not allowed by Let's Encrypt))
                  • certbot ... --rsa-key-size 2048 ...
                2. upload the certificate to IAM (Upload the Certificate):
                  • aws iam upload-server-certificate --server-certificate-name ${domain}.cert \
                    --certificate-body file:///etc/letsencrypt/live/${domain}/cert.pem \
                    --private-key file://                    /etc/letsencrypt/live/${domain}/privkey.pem \
                    --certificate-chain file://                    /etc/letsencrypt/live/${domain}/chain.pem
                3. Get the ARN of the uploaded certificate
                  • ARN=$(aws --output json iam get-server-certificate --server-certificate-name ${domain}.cert | jq '.ServerCertificate.ServerCertificateMetadata.Arn')
                4. Add an HTTPS Listener Using the AWS CLI
                  • aws elb create-load-balancer-listeners --load-balancer-name my-load-balancer --listeners Protocol=HTTPS,LoadBalancerPort=443,InstanceProtocol=HTTPS,InstancePort=443,SSLCertificateId=$ARN
                5. ...
            • Renewal
              • letsencrypt-aws
                • Installation
                  • git clone https://github.com/alex/letsencrypt-aws.git
                  • cd letsencrypt-aws
                  • virtualenv env
                  • source env/bin/activate
                  • pip install -r requirements.txt
                  • deactivate
                • First time
                  • ask ACME for a private key
                    • source env/bin/activate
                    • python letsencrypt-aws.py register email@host.com
                    • deactivate
                    • save it as email_host_com.privkey.acme.pem (locally or on AWS S3)
                  • create an IAM policy (name it e.g. LetsencryptAWS)
        • Problemes / Problems
          • The requested nginx plugin does not appear to be installed? #1736
          • Solution: manual request and installation
            • cd letsencrypt
            • ./letsencrypt-auto certonly -a manual --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • /etc/nginx/nginx.conf (or /etc/nginx/conf.d/your_site.conf)
              • ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
                ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
            • Note: if you specify cert.pem instead of fullchain.pem, browsers will work, but curl nor openssl s_client won't:
    • Content negotiation
    • JPEG JFIF
    • SOAP (Simple Object Access Protocol)
    • Platform for Privacy Preferences (P3P) (press release)
    • Resource Description Framework (wp) (metadades/metadata)
    • Media Fragments URI (MFWG)
    • WebVTT (Web Video Text Tracks)
    • Cross-Origin Resource Sharing (CORS)
      • Access-Control-Allow-Origin: http://foo.example/
    • W3C TV
  • IETF RFC
  • Protocols de missatgeria / Messaging protocols

    		• broker implementation (MOM) clients


      		Android
      AMQP
      MQTT
      STOMP
    • AMQP (Advanced Message Queuing Protocol) (wp)
    • MQTT (Message Queue Telemetry Transport) (wp)
    • STOMP (Streaming Text Oriented Messaging Protocol) (wp)
      • to connect to message-oriented middleware
      • STOMP Protocol Specification, Version 1.2
      • Frames and headers

        		 frame standard headers
        		 request headers response headers


        		content-length content-type receipt

        connection frames CONNECT MAY

        • accept-version
        • host
        • login
        • passcode
        • heart-beat
        CONNECTED MAY


        • version
        • heart-beat
        • session
        • server
        client frames SEND MAY SHOULD MAY
        • destination
        • transaction

        SUBSCRIBE MAY
        		MAY
        • destination
        • id
        • ack: [auto, client, client-individual]
        UNSUBSCRIBE MAY
        		MAY
        • id

        BEGIN MAY
        		MAY
        • transaction
        COMMIT MAY
        		MAY
        • transaction

        ABORT MAY
        		MAY
        • transaction

        ACK MAY
        		MAY
        • id
        • transaction

        NACK MAY
        		MAY
        • id
        • transaction

        DISCONNECT MAY
        		MAY
        • receipt

        server frames MESSAGE MAY SHOULD

        • destination
        • message-id
        • subscription
        • ack
        RECEIPT MAY


        • receipt-id
        ERROR MAY SHOULD

        • receipt-id
        • message
      • Server
      • Client
        • CLI
          • stomp (from stomp.py)
            • stomp --verbose --ssl -H b-xxxx-1.mq.eu-south-2.amazonaws.com -P 61614 --user=myuser --password=mypassword
              • subscribe /queue/my_queue
              • ...
            • stomp --verbose -H vis.musicradio.com -P 61613
              • subscribe /topic/fm/ce1/c479/09580/image
              • ...
          • telnet/ncat client (producer: SEND; consumer: SUBSCRIBE)
            • RadioDNS
            • Notes
              • to end every frame:
                • ^@ = Ctrl+@
              • to end communication:
                • when using ncat:
                  • (just DISCONNECT)
                • when using telnet:
                  • ^] = Ctrl+]
            • frame (you must put an empty line between header and body and end the frame with ^@)
              • COMMAND
                header1:value1
                header2:value2

                Body^@
            • telnetncat vis.musicradio.com 61613
              • Trying 81.20.48.151...
                Connected to 81.20.48.151.
                Escape character is '^]'.
              • CONNECT

                ^@
              • SUBSCRIBE
                destination: /topic/fm/ce1/c479/09580/image
                ack: auto

                ^@
              • DISCONNECT
                receipt:77

                ^@
        • Python
  • HTTP headers
  • HTTP request methods (wp)
    • method
      		 description
      		 note
      		 usage
      		 related HTML tag



      		safe
      		 idempotent
      HEAD
      		 same as GET, but without the body



      GET
      		 requests a representation of the resource
      		 only to retrieve
      		 y
      		 y
      		 href
      POST
      		 submits data to be processed
      		n
      		 n
      		 form method="post"
      PUT
      		 uploads a representation of the resource
      		n
      		 y
      DELETE
      		 deletes the resource



      TRACE
      		 echoes the received request



      OPTIONS
      		 http methods supported by the server



      CONNECT




      PATCH
      		 apply partial modifications to a resource



    • Ús / Usage
    • curl -X http_method
    • Django
    • POST
      • generation
        		 received by CGI
        html form
        		 curl
        		 content-type
        		 as standard input
        <form action="my_cgi.sh" method="post">
        <input type="text" name="tata" value="tata_value">
        <input type="text" name="titi" value="titi_value">
        <input type="submit">
        </form>
        • curl -i -X POST --data 'tata=tata_value' --data 'titi=titi_value'
        • curl -i -H "Content-Type: application/x-www-form-urlencoded" -d 'tata=tata_value&titi=titi_value'
        • curl -i -X POST --data-binary 'tata=tata_value' --data-binary 'titi=titi_value'
        		 application/x-www-form-urlencoded
        		 tata=tata_value&titi=titi_value
        <form action="my_cgi.sh" method="post" enctype="multipart/form-data">
        <input type="text" name="tata" value="tata_value">
        <input type="text" name="titi" value="titi_value">
        <input type="submit">
        </form>
        • curl -i -X POST -F tata=tata_value -F titi=titi_value
        • curl -i -X POST -H 'Content-Type: multipart/form-data' -F tata=tata_value -F titi=titi_value
        		 multipart/form-data; boundary=------------------------6e0b5ea0578a6399
        		 --------------------------6e0b5ea0578a6399
        Content-Disposition: form-data; name="tata"

        tata_value
        --------------------------6e0b5ea0578a6399
        Content-Disposition: form-data; name="titi"

        titi_value
        --------------------------6e0b5ea0578a6399--

        • curl -i -X POST -H 'Content-Type: application/json' --data '{"toto":"toto_value","tata":"tata_value"}'
        • url="https://127.0.0.1:8000/"
          my_var="toto value";
          json_code="{\"toto\":\"${my_var}\"}";
          curl_options="          -H 'Content-Type: application/json' --data '$json_code'";
          curl_command="curl -v -X POST $curl_options '$url'";
          eval "$curl_command"
        • my_var="toto l'educat"; my_escaped_var=$(echo "${my_var}" | sed "s/'/'\\\''/g");
          json_code="{\"toto\":\"${my_escaped_var}\"}";
          curl_options="          -H 'Content-Type: application/json' --data '$json_code'";
          curl_command="curl -v -X POST $curl_options '$url'";
          eval "$curl_command"
        		application/json
  • HTTP status codes
    • Response Status Codes (RFC 7231, Section 6)
    • Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) (RFC 2324)
      • 418
    • List of HTTP status codes (wp)
    • Setting HTTP Status Codes (Servlet tutorial)
    • family
      		 description
      		 code
      1xx
      		 Informational
      • 100 Continue
      • 101 Switching Protocols
      • 102 Processing
      • 103 Early Hints
      2xx
      		 Success
      • 200 OK
      • 201 Created
      • 202 Accepted
      • ...
      3xx
      		 Redirection
      • 300 Multiple Choices
      • 301 Moved Permanently (POST is switched to GET)
      • ...
      • 308 Permanent Redirect (do not switch methods)
      4xx
      		 Client error
      • 400 Bad Request
      • 401 Unauthorized
      • 402 Payment Required
      • 403 Forbidden
      • 404 Not Found
      • 405 Method Not Allowed
      • ...
      • 408 Request Timeout
      • ...
      • 409 Conflict
      • ...
      • 415 Unsupported Media Type
      • ...
      • 418 I'm a teapot
      • ...
      • 429 Too Many Requests
      • ...
      • 451 Unavailable For Legal Reasons
      • ...
      • 460 The load balancer received a request from a client, but the client closed the connection with the load balancer before the idle timeout period elapsed.
      5xx
      		 Server error
      • 500 Internal Server Error
      • 501 Not Implemented
      • 502 Bad Gateway
      • 503 Service Unavailable
      • 504 Gateway Timeout
      • 505 HTTP Version Not Supported
      • ...
    • 4xx
  • DOCTYPE / DTD 
  • Emmagatzematge / Storage
  • Same origin policy (wp)
  • JSON-RPC
    • Info
    • Basics
      • request frame:
        • method attribute (mandatory)
        • params attribute (optional)
      • if request frame has an optional id attribute, then a server must send a response frame
      • frames without id are considered notification frames; server does not response on notification frames
      • communication is bidirectional
    • Over
  • Push
    • Comet / Ajax push (wp)
    • WebSocket (wp) (HTML5)
      • RFC 6455
      • WebSocket in Django
      • Websockets load-balancing with HAProxy
      • 9 Websocket Servers for Reliable Real-time Applications
      • WebSocket (Full Stack Python)
        • arquitectura
      • Escalabilitat / Scalability
      • Eines / Tools
        • nginx
        • ...
          format server client
          raw CLI
          Languages
          		 CLI
          • wscat -c ws://127.0.0.1:3000
          • websocat ws://127.0.0.1:3000
          • iocat ws://127.0.0.1:3000
          • Weasel
          Languages
          JSON-RPC
          		Languages
          socket.io CLI
          • iocat --socketio -l -p 3000
          Languages
          		 CLI
          • iocat --socketio ws://127.0.0.1:3000
            • as iocat is using ?EIO=3, error "Unsupported protocol version" is returned; to solve this, new (v4) socket.io server has to be started with option:
              • js:
                • const io = new Server(server, {
                      allowEIO3: true // false by default
                  });
              • python: ...
          • iocat --emit-key "chat message" --socketio ws://127.0.0.1:3000/
          Languages

        		• info instal·lació / installation docs servidor / server client
          Firefox



          		Responses
          • raw
          • JSON
          • Socket.IO
          • ...
          Weasel




          websocat
          • Rust toolchain (Mageia)
            • curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
            • cargo install --features=ssl websocat
          • ...
          • websocat -s 1234
          • websocat ws://127.0.0.1:1234/
          • websocat ws://ws.vi-server.org/mirror
          • websocat --jsonrpc ...
          wscat
          		npm install -g wscat

          • wscat -c ws://websocket-echo.com
          ws
          • plain



          socket.io
          • bash
          • JavaScript (Node.js)
            • socket.io
              • import { Server } from "socket.io";

                const io = new Server(3000);

                io.on("connection", (socket) => {
                    // send a message to the client
                    socket.emit("hello", "world");

                    // receive a message from the client
                    socket.on("howdy", (arg) => {
                        console.log(arg); // prints "stranger"
                    });
                });
              • import { createServer } from "http";
                import { Server } from "socket.io";

                const httpServer = createServer();
                const io = new Server(httpServer, {
                  path: "/my-custom-path/"
                });
              • const express = require('express');
                const app = express();
                const http = require('http');
                const server = http.createServer(app);
                const { Server } = require("socket.io");
                const io = new Server(server);

                app.get('/', (req, res) => {
                  res.sendFile(__dirname + '/index.html');
                });

                io.on('connection', (socket) => {
                    console.log('a user connected');
                    socket.on('disconnect', () => {
                        console.log('user disconnected');
                    });
                    socket.on('chat message', (msg) => {
                        io.emit('chat message', msg);
                    });
                });

                server.listen(3000, () => {
                  console.log('listening on *:3000');
                });
              • ...
          • Python
            • python-socketio
              • from aiohttp import web
                import socketio

                sio = socketio.AsyncServer()
                app = web.Application()
                sio.attach(app)

                async def index(request):
                    """Serve the client-side application."""
                    with open('index.html') as f:
                        return web.Response(text=f.read(), content_type='text/html')

                @sio.event
                def connect(sid, environ):
                    print("connect ", sid)

                @sio.event
                async def chat_message(sid, data):
                    print("message ", data)
                    await sio.emit("chat_message", data)

                @sio.event
                def disconnect(sid):
                    print('disconnect ', sid)

                app.router.add_static('/static', 'static')
                app.router.add_get('/', index)

                if __name__ == '__main__':
                    web.run_app(app)
            • deployment creation
              aiohttp from aiohttp import web
              import socketio

              sio = socketio.AsyncServer()
              app = web.Application()
              sio.attach(app)

              async def index(request):
                  """Serve the client-side application."""
                  with open('index.html') as f:
                      return web.Response(text=f.read(), content_type='text/html')

              @sio.event
              ...

              app.router.add_static('/static', 'static')
              app.router.add_get('/', index)

              if __name__ == '__main__':
                  web.run_app(app)
              WSGI import socketio

              sio = socketio.Server()
              app = socketio.WSGIApp(sio)
              ASGI import socketio

              sio = socketio.AsyncServer(async_mode="asgi")
              app = socketio.ASGIApp(sio)               # not needed when using Daphne
              # uvicorn.run(app, host='127.0.0.1', port=5000)


            • ...
          • ...

          • bash
          • JavaScript (browser, Node.js, React Native)
            • socket.io-client
              • <!DOCTYPE html>
                <html>
                  <head>
                  </head>
                  <body>
                    <script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
                    <script>
                      import {io} from "socket.io-client";

                      const socket = io("ws://localhost:3000")

                      // receive a message from server
                      socket.on("hello", (arg) => {
                      console.log(arg); // prints "world"
                      });

                      // send a message to the server
                      socket.emit("howdy", "stranger");
                    </script>
                  </body>
                </html>
              • import { io } from "socket.io-client";

                const socket = io("https://example.com", {
                  path: "/my-custom-path/"
                });
              • ...
                  <body>
                    <ulid="messages"></ul>
                    <form id="form" action="">
                      <input id="input" autocomplete="off" /><button>Send</button>
                    </form>
                   
                    <script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
                    <script>
                      var socket = io();

                      var messages = document.getElementById('messages');
                      var form = document.getElementById('form');
                      var input = document.getElementById('input');

                      form.addEventListener('submit', function(e) {
                          e.preventDefault();
                          if (input.value) {
                              socket.emit('chat message', input.value);
                              input.value = '';
                          }
                      });

                      socket.on('chat message', function(msg) {
                          var item = document.createElement('li');
                          item.textContent = msg;
                          messages.appendChild(item);
                          window.scrollTo(0, document.body.scrollHeight);
                      });
                    </script>   
                  </body>
                </html>
              • ...
          • Python
            • python-socketio
              • import asyncio
                import socketio

                sio = socketio.AsyncClient()

                @sio.event
                async def connect():
                    print("Received event: connect")
                    print("my sid is", sio.sid)

                @sio.event
                def connect_error(data):
                    print("The connection failed!")

                @sio.on('*')
                async def catch_all(event, data):
                   print(f"Received event {event} with data: {data}")

                @sio.event
                async def my_message(data):
                    print("Received event: my_message, with data: ", data)
                    await sio.emit('my response', {'response': 'my response'})

                @sio.on("chat message")
                async def on_missatge(data):
                    print("Received event: chat message, with data: ", data)

                @sio.event
                async def disconnect():
                    print("Received event: disconnect")

                async def main():
                    await sio.connect('http://localhost:3000')
                    await sio.wait()

                if __name__ == '__main__':
                    asyncio.run(main())
            • ...
          • ...
  • Media types (ftp)
  • Byte articles
  • Cache
  • vCard and vCalendar
  • MIME-Types
  • Imatges / Images

Proxy

  • Info
    • Proxy vs reverse proxy
      • [direct] proxy: una petició iniciada des d'un client en una subxarxa privada cap a Internet pot passar per un proxy, que va a buscar la resposta a Internet i li retorna cap al client
        • exemples: Squid, ...
      • reverse proxy: una petició iniciada des d'un client a Internet cap a un servidor, passa per un proxy invers, que va a buscar la resposta en un altre servidor (normalment en una subxarxa interna) i li retorna la resposta cap al client a Internet
        • tasques: balanceig de càrrega, autenticació, desxifratge, cache
        • exemples: nginx, ...
  • General
  • Articles
  • Programari / Software
  • Server
  • Client
    • configuració / setup
      • bash environment variable
        • export http_proxy=http://server-ip:port/
        • export http_proxy=http://USERNAME:PASSWORD@server-ip:port/
      • GUI
        • Centre de control de Mageia
          • Xarxa i internet
            • Servidor intermediari
    • ús / usage
      • CLI
        • wget http://toto.org/page.html
        • wget --proxy-user=USERNAME --proxy-password=PASSWORD http://toto.org/page.html
        • curl http://toto.org/page.html
        • curl --proxy-user user:password http://toto.org/page.html
      • aplicacions / applications
        • Firefox
          • Preferències -> Paràmetres de xarxa
        • Chrome
          • Configuració -> Sistema -> Obre la configuració del servidor intermediari de l'ordinador
  • Discover

Servidors HTTP / HTTP servers

  • Comparison of web servers (wp)
  • Comparison of lightweight web servers (wp)
  • Best Practices for Speeding Up Your Web Site (Steve Sounders)
  • Servidors HTTPS / HTTPS server
  • Balanceig de càrrega / Load balancing
  • Cache on servers
  • Servidors de prova / Test servers
  • Nginx (nginx.com) (wp)
    • Compilació / Compilation
      • Init scripts
      • show compilation options
        • nginx -V
      • check config
        • nginx -t
    • Instal·lació / Installation
      • Mageia
        • urpmi nginx
        • sytemctl enable nginx
        • sytemctl start nginx
      • Ubuntu
      • CentOS
        • How To Install Nginx on CentOS 7
        • LEMP server on CentOS 7 with FastCGI
        • sudo yum install epel-release
        • sudo yum install nginx
        • Official nginx repositories for CentOS
          • /etc/yum.repos.d/nginx.repo
        • sudo systemctl start nginx.service
        • Firewall (only needed if firewalld.service is running)
          • sudo firewall-cmd --permanent --zone=public --add-service=http
            sudo firewall-cmd --permanent --zone=public --add-service=https
            sudo firewall-cmd --reload
        • SELinux
          • SELinux on HTTP servers
          • sudo setsebool -P httpd_read_user_content 1
          • if only using nginx (not uwsgi)
          • if using also uwsgi (unix socket permission denied)
            • SELinux policy for nginx and GitLab unix socket in Fedora 19
            • disable Selinux:
              • setenforce 0
            • detect the problem
              • access your website, to generate logs
              • sudo yum install policycoreutils-python
              • sudo grep nginx /var/log/audit/audit.log | audit2allow
            • solve the problem (do not use -M option, as you will not be able to edit the te text file to add rules)
              • sudo -i
              • grep nginx /var/log/audit/audit.log | audit2allow -m nginx > nginx.te
              • checkmodule -M -m -o nginx.mod nginx.te
              • semodule_package -o nginx.pp -m nginx.mod
              • semodule -i nginx.pp
              • setenforce 1
            • example
              • nginx.te
                • module nginx 1.0;

                  require {
                          type httpd_t;
                          type init_t;
                          type nfs_t;
                          type user_home_t;
                          type var_lib_t;
                          type unlabeled_t;
                          class unix_stream_socket connectto;
                          class file {read getattr open};
                          class lnk_file read;
                          class sock_file write;
                  }

                  #============= httpd_t ==============
                  allow httpd_t init_t:unix_stream_socket connectto;
                  allow httpd_t nfs_t:file {read getattr open};
                  allow httpd_t nfs_t:lnk_file read;
                  allow httpd_t user_home_t:file {read open};
                  allow httpd_t var_lib_t:sock_file write;
                  allow httpd_t unlabeled_t:lnk_file read;
                  allow httpd_t unlabeled_t:file {read open};
          • Problems / Problemes
      • Django wsgi
    • Documentation (.org)
    • Service setup
      • NGINX systemd service file
      • nginx.service
        • # https://www.nginx.com/resources/wiki/start/topics/examples/systemd/

          [Unit]
          Description=The NGINX HTTP and reverse proxy server
          After=syslog.target network.target remote-fs.target nss-lookup.target cloud-init.service

          [Service]
          Type=forking
          PIDFile=/run/nginx.pid
          TimeoutStartSec=200s
          ExecStartPre=/usr/sbin/nginx -t
          PrivateTmp=true

          # other ExecStartPre scripts
          ExecStartPre=/bin/bash -c 'for script in /etc/nginx/execstartpre.d/*; do $script; done'

          ExecStart=/usr/sbin/nginx
          ExecReload=/bin/kill -s HUP $MAINPID
          ExecStop=/bin/kill -s QUIT $MAINPID

          [Install]
          WantedBy=multi-user.target
      • /tmp
        • /usr/lib/systemd/system/nginx.service
        • real /tmp dir for nginx is:
          • /tmp/systemd-private-...-nginx.service-.../tmp
    • Configuració / Configuration
      • How To Optimize Nginx Configuration
        • Timeouts
      • Modificació del fitxer de configuració / Modification of config file
      • Variables
      • Modular




        • worker_processes number | auto;


          events { ... } worker_connections number;

          # ngx_http_core_module
          http { ... }          		 include file | mask; include mime.types;
          default_type application/octet-stream;
          sendfile on;
          keepalive_timeout  65;
          gzip on;


          		# ngx_http_upstream_module
          upstream name { ... }           		# round-robin
          server address [parameters];

          server 127.0.0.1:8001;
          server 127.0.0.1:8002;

          		# other server{} sections in files:
          include /etc/nginx/conf.d/*.conf;


          		server { ... } listen address[:port] ...;
          listen port ...;
          listen unix:path ...;

          server_name name ...; #           see: regex
          listen 80;
          server_name localhost;
          access_log           /var/log/nginx/proxy.access.log;
          error_log /var/log/nginx/proxy.error.log;


          		# ngx_http_ssl_module
          ssl_certificate file;
          ...

          ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
          ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
          ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
          ssl_ciphers         HIGH:!aNULL:!MD5;


          		# nginx-vod-module: vod settings
          vod_mode ...;
          vod_upstream_location ...;
          vod_fallback_upstream_location ...;
          vod_last_modified ...;
          vod_last_modified_types ...;

          # nginx-vod-module: vod caches
          ...

          # nginx-vod-module: file handle caching / aio
          ...


          		# other location{}
          include ...*.conf;


          		location [ = | ~ | ~* | ^~ ] uri { ... }
          • = exact matching
          • ~ case-sensitive matching regex
          • ~* case-insensitive matching regex
          • ^~ ...
          location @name { ... }

          location / { ... }           		root ...;

          root html;



          		alias ...;



          		# ngx_http_proxy_module
          proxy_pass ...;
          proxy_read_timeout ...;
          proxy_send_timeout ...;
          proxy_http_version ...;
          proxy_set_header ...;
          proxy_redirect ...;



          		# ngx_http_fastcgi_module
          fastcgi_pass ...;
          fastcgi_param ...;



          		# ngx_http_uwsgi_module
          uwsgi_pass ...;


          		location = /50x.html {} root html;
          # nginx-rtmp-module
          rtmp{}          		 include /etc/nginx/rtmp.conf.d/*.conf;


          		server{} listen 1935;


          		application ... {} live on;
          wait_video on;
          wait_key on;
        • Regex (PCRE - Perl regular expressions)


          		• examples
            server Wildcard names
            • server_name .example.org;
              • matches: example.org, *.example.org
            asterisk only at the begin or end:
            • server_name *.example.org;
            • server_name my.example.*;
            asterisk not at the begin nor end is not allowed:
            • www.*.example.org (use regex instead: ~^www\..+\.example\.org$)
            • w*.example.org (use regex instead: ~^w.*\.example\.org$)
            Regular expressions names
            (start with ~)
            • server_name  ~^www\d+\.example\.net$;
            • a regular expression containing the characters “{” and “}” should be quoted:
              • server_name  "~^(?<name>\w\d{1,3}+)\.example\.net$";
            • digital captures
              • usage:
                • server {
                      server_name   ~^(www\.)?(.+)$;

                      location / {
                          root   /sites/$2;
                      }
                  }
            • named regular expression
              • syntax:
                • ?<name> Perl 5.10 compatible syntax, supported since PCRE-7.0
                • ?'name' Perl 5.10 compatible syntax, supported since PCRE-7.0
                • ?P<name> Python compatible syntax, supported since PCRE-4.0
              • usage:
                • server {
                      server_name   ~^(www\.)?(?<domain>.+)$;

                      location / {
                          root   /sites/$domain;
                      }
                  }
            • ...
            location Regular expressions
            • (empty) starting with
            • = exact matching
            • ~ case-sensitive matching regex
            • ~* case-insensitive matching regex
            • ^~ ...
            • location = / {
                  [ configuration A ]
              }
            • location / {
                  [ configuration B ]
              }
            • location /documents/ {
                  [ configuration C ]
              }
            • location ^~ /images/ {
                  [ configuration D ]
              }
            • location ~* \.(gif|jpg|jpeg)$ {
                  [ configuration E ]
              }
          • ...
        • /etc/nginx/
          • nginx.conf
            • user  nginx;
              worker_processes  1;

              http {
                  include       /etc/nginx/mime.types;
                  default_type  application/octet-stream;
                  # Load modular configuration files from the /etc/nginx/conf.d directory.
                  # See http://nginx.org/en/docs/ngx_core_module.html#include
                  # for more information.
                  include /etc/nginx/conf.d/*.conf;

              }

              # nginx-rtmp-module
              rtmp {
                  # Load modular configuration files from the /etc/nginx/rtmp.conf.d directory.
                  include /etc/nginx/rtmp.conf.d/*.conf;
              }
          • execstartpre.d/
            • script1_execstartpre.sh
            • ...
          • conf.d/
            • http_server_1.conf
              • server {
                    listen       80;
                    server_name  localhost;

                    # Load modular configuration files from the /etc/nginx/conf.d/server_1.d/ directory.
                    include /etc/nginx/conf.d/http_server_1.d/*.http_server.conf;
                }
            • http_server_1.d/
              • dir_1_a.http_server.conf
                • location /dir_1_a {
                      alias /path/to/dir_1_a;
                  }
              • rtmp_server_1.http_server.conf (if using nginx-rtmp-module statistics and control)
                • # rtmp statistics location /stat {
                      rtmp_stat all;
                      rtmp_stat_stylesheet stat.xsl;
                  }

                  location /stat.xsl {
                      root /path/to/parent/dir/of/stat/xsl/file;
                  }

                  # rtmp control
                  location /control {
                      rtmp_control all;
                  }
            • https_server_2.conf
              • server {
                    # https
                    listen              443 ssl;
                    server_name         www.example.org;

                    ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
                    ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
                    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
                    ssl_ciphers         HIGH:!aNULL:!MD5;
                    ...
                }
            • https_server_2.d/
              • ...
          • rtmp.conf.d
            • rtmp_server_1.conf
              • server {
                    listen 1935;
                    ping 30s;
                    notify_method get;

                    # compatibility with GStreamer:
                    publish_time_fix off;

                    # Load modular configuration files from the /etc/nginx/rtmp.conf.d/wct-rtmp/ directory.
                    include /etc/nginx/rtmp.conf.d/rtmp_server_1.d/*.rtmp_server.conf;
                }
            • rtmp_server_1.d/
              • app_1_a.rtmp_server.conf
                • application app_1_a {
                      live on;
                      wait_video on;
                      wait_key on;
                  }
            • rtmp_server_2.conf
              • ...
            • rtmp_server_2.d/
              • ...
      • HTTPS
    • Autenticació / Authentication
    • Debug
      • A debugging log (nginx.org)
        • Debugging log for selected clients
        • Logging to a cyclic memory buffer
      • Debugging NGINX (nginx.com)
      • when building
        • ./configure --with-debug ...
        • check it:
          • nginx -V 2>&1 | grep -- '--with-debug'
      • nginx.conf
        • error_log /path/to/log debug;
        • error_log /var/log/nginx/error.log debug;
      • level possible values: debug, info, notice, warn, error, crit, alert, emerg
      • tail -n 200 -f /var/log/nginx/error.log
    • Logs
      • Logging and monitoring
      • Error log
      • Access log
        • log_format
          • default nginx.conf
            • http {
              ...
                  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for"';
              ...
              }
          • add request_time:
            • http {
              ...
                  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for" $request_time';
              ...
              }
        • access_log
      • access.log columns when using awk BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; :
        • $0 all line
          $1 IP address
          $2 -
          $3 -
          $4 datetime
          $5 timezone
          $6 request
          $7 response code
          $8 bytes
          $9 referer
          $10 user_agent
          $11
          		 x_forwarded_for (real origin ip address)
          $12
          		 (request_time)
      • count the most requested urls:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; {print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
      • count the most requested urls inside a specified time period:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; $4 ~ /14\/Dec\/2021:13:5/ {print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -n
      • get requests with code 4xx and print time, code, request:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; $4 ~ /17\/Dec\/2021/ && $7 ~ /^4/ {print $4,$7,$6}' /var/log/nginx/access.log
    • Monitoratge / Monitoring
      • ngxtop
        • Instal·lació / Installation
          • pip install ngxtop
        • Ús / Usage
          • only last entries (live)
            • ngxtop
          • all log content
            • ngxtop --no-follow -i 'status >= 500' print request status http_referer
    • Headers
      • your.conf
        •     location /toto {
                  add_header x-peticio $request;
                  add_header x-temps $request_time;
                  add_header x-longitud $request_length;
                  return 200 'tot bé\n';
              }
    • Reverse proxying to:
      • Django deployment
      • Nginx: the High-Performance Web Server and Reverse Proxy
      • nginx config backend server
        		 gateway interface apps

        		memcached



        # round robin
        upstream python_servers {
          server 127.0.0.1:8001;
          server 127.0.0.1:8002;
        }
        server {
          listen   127.0.0.1:8000;
          server_name  proxy;
          access_log  /var/log/nginx/proxy.access.log;
          error_log /var/log/nginx/proxy.error.log;
          location / {
            proxy_pass http://python_servers;
          }
        }         		ngx_http_proxy_module
        • server.py
          • import sys,BaseHTTPServer as B
            class Handler(B.BaseHTTPRequestHandler):
              def do_GET(self):
                self.wfile.write("Served from port %s" % port)
              def log_message(self, *args):
                pass
            if __name__ == '__main__':
              host,port = sys.argv[1:3]
              server = B.HTTPServer((host,int(port)), Handler)
              server.serve_forever()
        • # python /tmp/server.py 127.0.0.1 8001 &
          # python /tmp/server.py 127.0.0.1 8002 &

        location /mywebsocket {
            proxy_read_timeout 1d;
            proxy_send_timeout 1d;
            proxy_pass http://localhost:8123;
            #proxy_pass http://unix:/tmp/unixsocket_websocat;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
        		websocket
        • websocat -s 8123
        • socket.io
          • node index.js
          • python server.py
            • sio = socketio.Server()
            • sio = socketio.AsyncServer()
        • ...


        		SCGI



        upload_pass ... upload-module



        location ~ \.php$ {
            include /etc/nginx/fcgi_params; #or whatever you named it
            fastcgi_pass  127.0.0.1:9000;
        }        		 FastCGI
        		PHP
        • php-cgi -b 127.0.0.1:9000
        		CGI
        server {
            ...
            location /myscript {
                fastcgi_pass unix:/var/run/fcgiwrap.socket;

                # Fastcgi parameters, include the standard ones
                include /etc/nginx/fastcgi_params;

                fastcgi_param SCRIPT_FILENAME /usr/local/bin/myscript.sh;
            }
        }        		 FCGIWrap
        • systemctl start spawn-fcgi


        # the upstream component nginx needs to connect to
        upstream django_mysite {
            server unix:///var/lib/uwsgi/mysite.sock;
        }

        # configuration of the server
        server {
            ...
            # Finally, send all non-media requests to the Django server.
            location / {
                uwsgi_pass django_mysite;
                include /etc/uwsgi/uwsgi_params
            }
        }         		UWSGI
        • bin/uwsgi
        		 WSGI
        • Django
        • Socket.IO WSGI
          • python wsgi_server.py
            • import socketio

              sio = socketio.Server()
              app = socketio.WSGIApp(sio)

        		Gunicorn

        upstream channels-backend {
             # use 127.0.0.1 instead of localhost to avoid errors related to ipv6
            server 127.0.0.1:9000;
        }

        server {
            ...
            location / {
                try_files $uri @proxy_to_app;
            }
          
            location @proxy_to_app {
                proxy_pass http://channels-backend;

                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";

                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $server_name;
            }
        }
        		 Daphne
        • daphne -p 9000 myproject.asgi:application
        • daphne -b 0.0.0.0 -p 9000 myproject.asgi:application
        • daphne -e ssl:443:privateKey=key.pem:certKey=crt.pem my_project.asgi:application
        • systemctl start my_django_project.daphne.service
          • [Unit]
            Description=Daphne daemon (my_project)
            After=network.target

            [Service]
            PIDFile=/run/daphne/pid
            User=root
            Group=root
            WorkingDirectory=/path/to/my_project
            ExecStart=/path/to/my_project/env/bin/daphne --bind 0.0.0.0 --port 9000 --verbosity 0 my_project.asgi:application
            ExecReload=/bin/kill -s HUP $MAINPID
            ExecStop=/bin/kill -s TERM $MAINPID
            Restart=on-abort
            PrivateTmp=true
            LimitNOFILE=32768

            [Install]
            WantedBy=multi-user.target
        • daphne -p 3000 asgi_server:app
        • daphne -b 0.0.0.0 -p 3000 asgi_server:app
        • systemctl start asgi_server.daphne.service
          • ...
        		ASGI
        • Django
          • my_project/my_project/asgi.py
            • import os
              import django
              from channels.routing import get_default_application

              os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")
              django.setup()
              application = get_default_application()
        • Socket.IO ASGI
          • python asgi_server.py
            • import socketio

              sio = socketio.AsyncServer()
              app = socketio.ASGIApp(sio)

        		Uvicorn

      • AWS CloudFront
      • FastCGI (CGI)
        • fastcgi module (ngx_http_fastcgi_module)
        • Understanding and Implementing FastCGI Proxying in Nginx
        • FastCGI example
        • fastcgi_params Versus fastcgi.conf – Nginx Config History
        • nginx - How to run a shell script on every request?
        • FCGIWrap (github)
          • Documentation
          • Installing FcgiWrap and Enabling Perl, Ruby and Bash Dynamic Languages on Gentoo LEMP
          • Installation
            • Ubuntu
              • sudo apt-get install fcgiwrap
              • (no need to install spawn-fcgi, as fcgiwrap already comes with a service)
              • sudo systemctl status fcgiwrap.service
            • Mageia
              • from repo
                • ...
              • from source
                • install dependencies
                  • sudo dnf install libfcgi-devel
                • download and compile
                  • git clone http://github.com/gnosek/fcgiwrap.git
                  • cd fcgiwrap
                  • autoreconf -i
                  • ./configure
                  • make
                  • sudo make install
                    • /usr/local/sbin/fcgiwrap
              • install and setup spawn-fcgi (needed?)
                • install
                  • sudo dnf install ...
                • setup
                  • ...
            • CentOS
              • from repo
                • sudo dnf install fcgiwrap
              • from source
              • install and setup spawn fcgi
                • install
                • setup
                  • /etc/sysconfig/spawn-fcgi
                    • # You must set some working options before the "spawn-fcgi" service will work.
                      # If SOCKET points to a file, then this file is cleaned up by the init script.
                      #
                      # See spawn-fcgi(1) for all possible options.
                      #
                      # Example :
                      #SOCKET=/var/run/php-fcgi.sock
                      #OPTIONS="-u apache -g apache -s $SOCKET -S -M 0600 -C 32 -F 1 -P /var/run/spawn-fcgi.pid -- /usr/bin/php-cgi"

                      FCGI_SOCKET=/var/run/fcgiwrap.socket
                      # if fcgiwrap was compiled from source
                      #FCGI_PROGRAM=/usr/local/sbin/fcgiwrap
                      # if fcgiwrap was installed from repo
                      FCGI_PROGRAM=                      /usr/sbin/fcgiwrap
                      FCGI_USER=nginx
                      FCGI_GROUP=nginx
                      FCGI_EXTRA_OPTIONS="-M 0770"
                      # IMPORTANT: -F 1 means only one worker!
                      OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/spawn-fcgi.pid -- $FCGI_PROGRAM"
                    • to redirect stderr logs to /var/log/nginx/error.log (they will be shown as "FastCGI sent in stderr:")
                      • ...
                        OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/spawn-fcgi.pid -- $FCGI_PROGRAM -f"
                • start
                  • systemctl enable spawn-fcgi
                  • systemctl start spawn-fcgi
                  • systemctl status spawn-fcgi
                  • ...
          • Problemes / Problems
          • Security issues
          • Example to execute a script specified in the url
            • setup
              • /etc/nginx/default.d/toto.conf
                • location /cgi-bin/ {
                       # Disable gzip (it makes scripts feel slower since they have to complete
                       # before getting gzipped)
                       gzip off;

                       # Set the root to /usr/lib (inside this location this means that we are
                       # giving access to the files under /usr/lib/cgi-bin)
                       root  /usr/share/nginx;

                       # Fastcgi socket
                       fastcgi_pass  unix:/var/run/fcgiwrap.socket;

                       # Fastcgi parameters: include the standard ones
                       include /etc/nginx/fastcgi_params;
                       #                   Fastcgi parameters: adjust non standard parameters (e.g. SCRIPT_FILENAME)
                       fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                       #                   Fastcgi parameters: user-defined parameters
                                         fastcgi_param TOTO_PARAM $request_length;
                  }
              • sudo systemctl reload nginx.service
            • scripts
              • mkdir -p /usr/share/nginx/cgi-bin
              • perl script
                • /usr/share/nginx/cgi-bin/hello_world.cgi
                  • #!/usr/bin/perl -w
                         # Tell perl to send a html header.
                         # So your browser gets the output
                         # rather then <stdout>(command line
                         # on the server.)
                    print "Content-type: text/html\n\n";
                         # print your basic html tags.
                         # and the content of them.
                    print "<html><head><title>Hello World!! </title></head>\n";
                    print "<body><h1>Hello world</h1></body></html>\n";
                • chmod 755 /usr/share/nginx/cgi-bin/hello_world.cgi
              • bash script
                • /usr/share/nginx/cgi-bin/toto.sh
                  • #!/bin/bash

                    # as PATH is set in /etc/init.d/functions, you may need to complete it
                    # when you are calling other scripts (e.g. in /usr/local/bin) from here
                    PATH=${PATH}:/usr/local/bin
                    export $PATH

                    # call a script in /usr/local/bin
                    my_script.sh

                    # content type must be present, followed by an empty line
                    # all lines must be ended with \r\n (CR LF)                     echo -e "Content-type: text/plain\r"
                    echo -e "\r"

                    # your text
                    echo "123"
                    echo "server software: $SERVER_SOFTWARE"
                    echo "script filename: $SCRIPT_FILENAME"
                    echo  "toto param: $TOTO_PARAM"
                    # print all available environment variables
                    env
                • chmod 755 /usr/share/nginx/cgi-bin/toto.sh
                • Debug
                  • /usr/share/nginx/cgi-bin/toto_debug.sh
                    • #!/bin/bash -x
                      # adding -x will make bash xtrace output (by default written to stderr) to appear in /var/log/nginx/error.log,
                      # if option -f was specified in spawn config

                      echo -e "Content-type: text/plain\n"

                      # log to stderr instead of response body
                      (1>&2 echo "[hello]")

                      ...
                • Parse GET and POST parameters (using cgi_functions.sh)
                  • See also CGI bash examples with POST
                  • /usr/share/nginx/cgi-bin/toto_with_params
                    • #!/bin/bash
                      # if cgi_functions.sh is installed in /usr/local/bin
                      PATH=/usr/local/bin:${PATH}
                      export PATH=$PATH
                      # register all GET and POST variables
                      source cgi_functions.sh
                      cgi_getvars BOTH ALL 1>&2

                      # content type must be present, followed by an empty line
                      # all lines must be ended with \r\n (CR LF)
                      echo -e "Content-type: text/plain\r"
                      echo -e "\r"

                      # variables from QUERY and POST are available as shell variables
                      echo "var1: $var1"
                      echo "var2: $var2"
                      echo "var3: $var3"
                  • GET
                    • curl -X GET http://127.0.0.1/cgi-bin/toto_with_params?var1=value1
                  • POST
                    • application/x-www-form-urlencoded
                      • curl -X POST --data 'var3=value3' http://127.0.0.1/cgi-bin/toto_with_params
                    • multipart/form-data
                      • curl -X POST -F var2=value2 http://127.0.0.1/cgi-bin/toto_with_params?var1=value1
                      • curl -X POST -F var2=value2 http://127.0.0.1/cgi-bin/toto_with_params
                  • from local:
                    • REQUEST_METHOD=POST QUERY_STRING="var1=value1&var4=value4" var2=value2 var3=value3 /usr/share/nginx/cgi-bin/toto_with_params
                    • echo "var2=value_a&var2=value_b&var3=value3" | (REQUEST_METHOD=POST CONTENT_TYPE=application/x-www-form-urlencoded CONTENT_LENGTH=1000 ./multi_cgi.sh)
                • Allow only POST:
                  • /usr/share/nginx/cgi-bin/toto_only_post
                    • #!/bin/bash # register all GET and POST variables
                      source cgi_functions.sh
                      cgi_getvars BOTH ALL 1>&2
                      if [[ $REQUEST_METHOD != "POST" ]]
                      then
                          echo "Status: 405 Method Not Allowed"
                          echo -e "Content-type: text/plain\r"
                          echo -e "\r"

                          echo "Method not allowed"
                          exit 0
                      fi                       # content type must be present, followed by an empty line # all lines must be ended with \r\n (CR LF)
                      echo -e "Content-type: text/plain\r"
                      echo -e "\r"

                      ...
            • Client
            • Problemes / Problems
              • SELinux
              • 403 Forbidden
                • chmod +x toto.sh
              • 502 Bad gateway
                • check that script returns as first lines (optionally a Status line, containing return code number and reason phrase, see: ngx_http_fastcgi_hide_headers) (note the empty line):
                  • Content-type: text/plain
                  • Status: 202 OK
                    Content-type: text/plain
          • Example to execute a fixed script when accessing a url:
            • nginx - How to run a shell script on every request?
            • ...conf with standard parameters
              • location /do_something/ {
                     # Disable gzip (it makes scripts feel slower since they have to complete
                     # before getting gzipped)
                     gzip off;
                     # Set the root to /usr/lib (inside this location this means that we are
                     # giving access to the files under /usr/lib/cgi-bin)
                     root  /usr/share/nginx;
                     # Fastcgi socket
                     fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                     # Fastcgi parameters, include the standard ones
                     include /etc/nginx/fastcgi_params;
                     # Adjust non standard parameters (SCRIPT_FILENAME)
                     fastcgi_param SCRIPT_FILENAME /path/to/fixed_script.sh;
                     # user-defined parameters
                                     fastcgi_param TOTO_PARAM $request_length;
                }
            • fixed_script.sh (see also simple_cgi.sh for an example with POST)
              • #!/bin/bash
                # content type must be present, followed by an empty line                 # all lines must be ended with \r\n (CR LF)
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                # execute some script
                result=$(/path/to/do_something_script.sh)
                echo "result:"
                echo $result

                # your text
                echo "request method:  $REQUEST_METHOD"
                echo "server software: $SERVER_SOFTWARE"
                echo "script filename: $SCRIPT_FILENAME"
                echo  "toto param: $TOTO_PARAM"
            • fixed_script_with_specified http response code
              • FastCGI and Nginx - Return HTTP Status
              • #!/bin/bash                                                                                                                                                                                                                       
                echo "Status: 404 Not Found"
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                echo "Uep, com anam?"
            • conf with parameters from url
              • location /fast {
                    # Disable gzip (it makes scripts feel slower since they have to complete
                    # before getting gzipped
                    gzip off;

                    # script will be under
                    root /usr/share/nginx;

                    # Fastcgi socket
                    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                    # Fastcgi parameters, include the standard ones
                    include /etc/nginx/fastcgi_params;

                    # script to be executed
                    fastcgi_param SCRIPT_FILENAME /usr/share/nginx/cgi-bin/fixed_script.sh;

                    # parameters for the script
                    fastcgi_param REQUEST_URI $request_uri;

                    # parameters from url
                    # /fast/<param1>/<param2>/
                    if ($request_uri                 ~* "/fast/([^/]*)/([^/]*)/$" ) {
                        set $param1 $1;
                        set $param2 $2;
                    }
                    fastcgi_param PARAM1 $param1;
                    fastcgi_param PARAM2 $param2;
                }
            • fixed_script with parameters
              • #!/bin/bash

                # content type must be present, followed by an empty line
                # all lines must be ended with \r\n (CR LF)
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                echo "REQUEST_URI: ${REQUEST_URI}"
                echo "PARAM1: ${PARAM1}"
                echo "PARAM2: ${PARAM2}"

                exit 0
          • Example to get upload time:
            • .../upload.conf
              • location /up {
                    # allow 100 Continue
                    client_max_body_size 20M;

                    # allow upload
                    sendfile on;

                    # Disable gzip (it makes scripts feel slower since they have to complete
                    # before getting gzipped)
                    gzip off;
                    # Set the root to /usr/lib (inside this location this means that we are
                    # giving access to the files under /usr/lib/cgi-bin)
                    root  /usr/share/nginx;
                    # Fastcgi socket
                    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                    # Fastcgi parameters, include the standard ones
                    include /etc/nginx/fastcgi_params;

                    # Adjust non standard parameters:
                    fastcgi_param SCRIPT_FILENAME /path/to/upload_time.sh;
                    # user-defined parameters:
                    fastcgi_param REQUEST_LENGTH $request_length;
                    fastcgi_param REQUEST_TIME $request_time;

                    # user-defined headers
                    add_header x-temps $request_time;
                    add_header x-longitud $request_length;                 }
            • upload_time.sh
              • #!/bin/bash
                # content type must be present, followed by an empty line
                # all lines must be ended with \r\n (CR LF)                 echo -e "Content-type: text/plain\r"
                echo -e "\r"

                # execute some script
                result=$(/path/to/do_something_script.sh)
                echo "result:"
                echo $result

                echo "request method:  $REQUEST_METHOD"
                echo "request length:  $REQUEST_LENGTH"
                echo "request time:    $REQUEST_TIME"
            • Client
              • dd if=/dev/zero of=megabyte.dat  bs=1M  count=1
              • curl -i --data-binary '@megabyte.dat' http://<server>/up
    • Cache
    • CORS
    • Upload
      • Test upload speed
      • Nginx direct file upload without passing them through backend
      • nginx upload client_max_body_size issue
        • Expect: 100-Continue
      • client
      • client_body_in_file_only
        • files uploaded to
          • /tmp/...
        • ...conf
          •  server {
                listen       80;
                server_name  localhost;

                error_log /var/log/nginx/error.debug.log debug;

                #sendfile on;

                location /up {
                  #auth_basic                 "Restricted Upload";
                  #auth_basic_user_file       basic.htpasswd;
                  #limit_except POST          { deny all; }

                  #client_body_temp_path      /tmp/;
                  #client_body_in_file_only   on;
                  #client_body_buffer_size    128K;                   # allow 100 Continue
                  client_max_body_size 20M;
                  #proxy_pass_request_headers on;
                  #proxy_set_header           X-FILE $request_body_file;
                  #proxy_set_body             off;
                  #proxy_redirect             off;
                  #proxy_pass                 http://www.example.org/;

                    sendfile on;

                    add_header x-peticio $request;
                    add_header x-temps $request_time;
                    add_header x-longitud $request_length;

                    #root html;
                    #return 200 '$request_body_file';

                    return 200 'tot bé\n';
                }
        • Client (POST)
          • curl -i --data-binary '@myfile.png' http://192.168.1.29/upload
        • Problemes / Problems
      • nginx-upload-module
        • Upload
        • Resumable upload
        • nginx-upload-module 2.2 (github)
        • Nginx upload module (v 2.2.0)
          • nginx.service
            • [Service]
              ...
              # create hashed dirs for upload-module (level1=1)
              ExecStartPre=/bin/sh -c '/bin/mkdir -p /var/www/uploads/{0..9}; chmod 777 -R /var/www/uploads/{0..9}'
          • Example configuration
              •     # allow 100 Continue.
                    # if not specified here, as general value, defaults to 1M, and limits upload,
                    #   even if a greater vallue is specified inside a location section
                    #   (which only controls return of 100 Continue message)
                    client_max_body_size 20M;
                    location /upload {
                        # this value would response 100 for files under 30M, but general client_max_body_size 20M
                        # would not allow upload of files between 20 and 30M:
                        #                 client_max_body_size 30M;

                        # Pass altered request body to this location
                        upload_pass   @new_upload;

                        # Store files to this directory
                        # The directory is hashed:
                        # level1=1 (digit) : subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
                        # (they can be created in nginx.service)
                        upload_store /var/www/uploads 1;
                       
                        # Allow uploaded files to be read only by user
                        upload_store_access user:r;

                        # Set specified fields in request body
                        # They are passed to backend as POST multipart/form-data
                        upload_set_form_field "${upload_field_name}_name" "$upload_file_name";
                        upload_set_form_field "${upload_field_name}_content_type" "$upload_content_type";
                        upload_set_form_field "${upload_field_name}_path "$upload_tmp_path";

                        # Inform backend about hash and size of a file
                        upload_aggregate_form_field "${upload_field_name}_md5" "$upload_file_md5";
                        upload_aggregate_form_field "${upload_field_name}_size" "$upload_file_size";

                        # pass fields from form
                        # pass all:
                        #upload_pass_form_field "(.*)";
                        # pass only submit and description:
                        upload_pass_form_field "^submit$|^description$";

                        upload_cleanup 400 404 499 500-505;
                    }

                    # Pass altered request body to a fastcgi backend                     location @new_upload {
                        # user-defined headers
                        add_header x-request-time $request_time;
                        add_header x-request-length $request_length;

                        fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                        # Fastcgi parameters, include the standard ones
                        include /etc/nginx/fastcgi_params;
                        # Adjust non standard parameters (SCRIPT_FILENAME)
                        fastcgi_param SCRIPT_FILENAME /usr/local/bin/new_upload.sh;

                        # user-defined parameters:
                        fastcgi_param REQUEST_LENGTH $request_length;
                        fastcgi_param REQUEST_TIME $request_time;
                    }
          • /usr/local/bin/new_upload.sh
            • #!/bin/bash
              echo -e "Content-type: text/plain\n"

              source cgi_functions.sh

              # register all GET and POST variables
              cgi_getvars BOTH ALL

              echo "New upload ==============================="
              export
        • Other examples
        • Compilation

        		•  client
          		 response code
          POST multipart/form-data curl -i -X POST -H 'Content-Type: multipart/form-data' -F toto='@myfile.png' http://<server>/up 200 OK
          POST x-www-form-urlencoded curl -i --data-binary '@myfile.png' http://<server>/up 415 Unsupported Media Type
          PUT
          		 curl --upload-file myfile.png http://<server>/up 405 Not Allowed
    • Resources (.com)
    • i18n
      • ...
    • 3rd party modules
    • Problemes / Problemes
      • check content of:
        • /var/log/nginx/error.log
        • /usr/local/nginx/logs/error.log
      • ...
      • related to fcgiwrap
        • connect() to unix:/var/run/fcgiwrap.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: ..., server: ..., request: ..., upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: ...
          • uwsgi with nginx (11: Resource temporarily unavailable) while connecting to upstream)
          • if, for some reason, a script called by fastcgi get stuck, next requests are not being served (netstats reports: CONNECTING), and they are queued.
          • This can be checked with: netstat --unix | grep fcgi | wc
          • is there a limit?
          • workaroud: systemctl restart spawn-fcgi.service
          • to get the process that is blocking fastcgi
          • debug:
            • netstat --unix
            • ss --unix
            • strace
            • lsof | grep fcgi
      • related to uWSGI (WSGI)
        • connect() to unix:///var/lib/uwsgi/....sock failed (11: Resource temporarily unavailable) while connecting to upstream
        • connection to page gives "502 Bad Gateway"
          • systemctl status emperor.uwsgi.service
            • import memcache
              ImportError: No module named memcache
        • connection to page gives "502 Bad Gateway"
          • /var/log/nginx/error.log
            • connect() to unix:///etc/uwsgi/wct_backend.sock failed (13: Permission denied) while connecting to upstream ...
          • If it doesn't work
          • Solució / Solution
            1. regenerate nginx.pp file
            2. sudo systemctl restart emperor.uwsgi.service
            3. sudo systemctl restart nginx.service
        • connection to page gives "502 Bad Gateway"
          • /var/log/nginx/error.log
            • connect() to unix:///etc/uwsgi/wct_backend.sock failed (2: No such file or directory) while connecting to upstream ...
          • Solució / Solution
            • check output from (maybe some misspelling?): 
              • sudo systemctl status emperor.uwsgi.service
            • check that uwsgi is installed in your virtualenv
            • check uwsgi configuration
      • related to Django and Daphne (ASGI)
        • connect() failed (111: Connection refused) while connecting to upstream, client: ..., server: ..., request: ..., upstream: "http://[::1]:9000/ws/...", host: ...
          • IPv6 not activated
          • Solució / Solution
            • conf.d/my_nginx.conf
              • upstream my-upstream {
                    # use 127.0.0.1 instead of localhost to avoid errors related to ipv6
                    server 127.0.0.1:9000;
                }
        • [alert] 2398#0: 1024 worker_connections are not enough
          • Optimal value for Nginx worker_connections
          • monitor the number of used worker_connections:
            • sudo ss -tupn | grep nginx | wc -l
            • (deprecated) sudo netstat -tupn | grep nginx | wc -l
          • Solució / Solution
            • nginx.conf
              • events {
                    # this value must not exceed:
                    # - max number of open files for nginx user and nginx.service:
                    #     sudo su - nginx -s /bin/bash -c "ulimit -Hn"
                    #     grep LimitNOFILE /usr/lib/systemd/system/nginx.service
                    # - total range of sockets to enable per IP: sysctl net.ipv4.ip_local_port_range
                    # default value: worker_connections  1024;
                    worker_connections  16384;
                }
            • set max number of open files
            • set total range of sockets to enable per IP
              • sysctl net.ipv4.ip_local_port_range
              • sysctl -w net.ipv4.ip_local_port_range ...
        • [error] 22590#0: *344 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: ..., server: ..., request: "GET ...", upstream: "http://127.0.0.1:9000/...", host: ...
        • [info] 22590#0: *355 peer closed connection in SSL handshake while SSL handshaking, client: ..., server: 0.0.0.0:443
          • ...
      • accept4() failed (24: Too many open files)
        • Nginx: 24: Too Many Open Files Error And Solution
        • nginx 24: Too many open files
        • see also: worker_connections are not enough
        • Solució / Solution:
          • increase limits for nginx user:
            • present values:
              • sudo su - nginx -s /bin/bash -c "ulimit -Hn"
              • sudo su - nginx -s /bin/bash -c "ulimit -Sn"
            • set new values at os level for user nginx:
              • /etc/security/limits.d/nginx.conf
                • # default: nginx       hard    nofile   4096
                  nginx       hard    nofile   16384
                  # default: nginx       soft    nofile   1024
                  nginx       soft    nofile   4096
            • set new values at service level:
              • /usr/lib/systemd/system/nginx.service
              • systemctl daemon-reload
              • systemctl restart nginx.service
            • check new values in running process:
              • nginx_pid=$(pgrep -f "nginx: master process")
              • cat /proc/${nginx_pid}/limits | grep -E "Limit|Max open files"
                • Limit                     Soft Limit           Hard Limit           Units    
                  Max open files            16384                16384                files  
          • perform stress tests
      • exec: fork failed (12: Cannot allocate memory), client: ..., server: ...
      • nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
        • Solució / Solution
          • sudo setsebool -P httpd_read_user_content 1
      • nginx: [emerg] bind() to 0.0.0.0:8000 failed (13: Permission denied)
      • nginx: [emerg] bind() to 0.0.0.0:444 failed (13: Permission denied)
      • content is not cached
  • Apache HTTP Server (Apache Software Foundation)
  • Fake http server with netcat
  • Cherokee Web Server (wp)
  • Netscape Enterprise Server 
  • NCSA HTTPd
  • CERN httpd
  • Servidors lleugers / Light servers
  • Jetty Java HTTP Servlet Server
  • Web Servers Comparison
  • Creating Net Sites
  • Designing Your Web Site
  • Databases interfaces
  • Utilitats / Utilities
  • Gateway interface
    gateway interface backend server config


    		nginx apache
    CGI (Common Gateway Interface) proxying to
    WSGI (Python Web Server Gateway Interface)
    • uwsgi
    • Gunicorn
    ASGI (Asynchronous Server Gateway Interface)
    • CGI (Common Gateway Interface)
      • CGI on Apache
      • FastCGI
      • RFC 3875 "The Common Gateway Interface (CGI) Version 1.1"
      • The Web Developer's Virtual Library course
      • CGIHTML (C routines)
      • CGI Developer's Guide
      • Bash
        • Bash.CGI
          • cgi_functions.sh
            • #!/bin/bash

              # https://oinkzwurgl.org/hacking/bash_cgi/
              # Phillippe Kehi <phkehi@gmx.net> and flipflip industries
              # to test this function locally:
              # source cgi_functions.sh
              # QUERY_STRING="var1=value1" cgi_getvars GET var1; echo $var1
              # QUERY_STRING="var1=value1&var2=value2" cgi_getvars GET ALL; echo $var1; echo $var2
              # CONTENT_TYPE="application/x-www-form-urlencoded" ... cgi_getvars POST ALL; echo $var1; echo $var2

              # to test this function locally:
              # source cgi_functions.sh
              # QUERY_STRING="var1=value1" cgi_getvars GET var1; echo $var1
              # QUERY_STRING="var1=value1&var2=value2" cgi_getvars GET ALL; echo $var1; echo $var2
              # CONTENT_TYPE="application/x-www-form-urlencoded" ... cgi_getvars POST ALL; echo $var1; echo $var2

              # (internal) routine to store POST data
              function cgi_get_POST_vars()
              {
                  # check content type
                  # FIXME: not sure if we could handle uploads with this..
                  if [ "${CONTENT_TYPE}" != "application/x-www-form-urlencoded" ] && ! [[ "${CONTENT_TYPE}" =~ ^"multipart/form-data" ]]
                  then
                      # if, in /etc/sysconfig/spawn-fcgi, OPTIONS contains -f,
                      # this echo can be seen at /var/log/nginx/error.log
                      echo "[`basename $0`] WARNING: received CONTENT_TYPE is ${CONTENT_TYPE}, but implemented content types for POST are only: "\
                           "application/x-www-form-urlencoded, multipart/form-data" 1>&2
                      return
                  fi

                  # save POST variables (only first time this is called)
                  [ -z "$QUERY_STRING_POST" \
                    -a "$REQUEST_METHOD" = "POST" -a ! -z "$CONTENT_LENGTH" ] && \
                      read -N $CONTENT_LENGTH RECEIVED_POST
                  #echo "RECEIVED_POST: $RECEIVED_POST"
                 
                  if [[ "${CONTENT_TYPE}" =~ ^"multipart/form-data" ]]
                  then
                      # export variables from multipart
                      boundary=$(echo $CONTENT_TYPE | awk -F= '{print$2}');
                      #QUERY_STRING_POST=$(echo "$RECEIVED_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS="&"} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print $1"="$3}')
                      # variables must be exported here, because they can contain ampersands in value
                      eval $(echo "$RECEIVED_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS=" "} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print "export "$1"=\""$3"\""}')
                  else
                      # take input string as is
                      QUERY_STRING_POST="$RECEIVED_POST"
                  fi
                 
                  return
              }

              # (internal) routine to decode urlencoded strings
              function cgi_decodevar()
              {
                  [ $# -ne 1 ] && return
                  local v t h
                  # replace all + with whitespace and append %%
                  t="${1//+/ }%%"
                  while [ ${#t} -gt 0 -a "${t}" != "%" ]; do
                      v="${v}${t%%\%*}" # digest up to the first %
                      t="${t#*%}"       # remove digested part
                      # decode if there is anything to decode and if not at end of string
                      if [ ${#t} -gt 0 -a "${t}" != "%" ]; then
                          h=${t:0:2} # save first two chars
                          t="${t:2}" # remove these
                          v="${v}"`echo -e \\\\x${h}` # convert hex to special char
                      fi
                  done
                  # return decoded string
                  echo "${v}"
                  return
              }

              # routine to get variables from http requests
              # usage: cgi_getvars method varname1 [.. varnameN]
              # method is either GET or POST or BOTH
              # the magic varible name ALL gets everything
              function cgi_getvars()
              {
                  [ $# -lt 2 ] && return
                  local q p k v s
                  # get query
                  case $1 in
                      GET)
                          [ ! -z "${QUERY_STRING}" ] && q="${QUERY_STRING}&"
                          ;;
                      POST)
                          cgi_get_POST_vars
                          [ ! -z "${QUERY_STRING_POST}" ] && q="${QUERY_STRING_POST}&"
                          ;;
                      BOTH)
                          [ ! -z "${QUERY_STRING}" ] && q="${QUERY_STRING}&"
                          cgi_get_POST_vars
                          [ ! -z "${QUERY_STRING_POST}" ] && q="${q}${QUERY_STRING_POST}&"
                          ;;
                  esac
                  shift
                  s=" $* "
                  # parse the query data
                  while [ ! -z "$q" ]; do
                      p="${q%%&*}"  # get first part of query string
                      k="${p%%=*}"  # get the key (variable name) from it
                      v="${p#*=}"   # get the value from it
                      q="${q#"$p"&*}" # strip first part from query string
                      # decode and assign variable if requested
                      [ -n "$p" ] && [ "$1" = "ALL" -o "${s/ $k /}" != "$s" ] && \
                          export "$k"="`cgi_decodevar \"$v\"`"
                  done
                  return
              }
        • CGI shell script using bash
        • Bash
          • Saving a binary file
        • How to parse $QUERY_STRING from a bash CGI script
        • Creating CGI Programs with Bash: Handling POST Data
          • curl -i -X POST --data 'tata=tata_value' http://127.0.0.1/test
        • Examples with POST
          • See also Parse GET and POST parameters
          • simple_cgi.sh
            • #!/bin/bash

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"
              echo -e '\r'

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"
              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"
              echo -e "QUERY_STRING: ${QUERY_STRING}\r"

              exit 0
            • #!/bin/bash

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"
              echo -e 'Access-Control-Allow-Origin:*\r'
              echo -e 'Access-Control-Allow-Credentials: true\r'
              echo -e 'Access-Control-Allow-Methods: GET, POST, OPTIONS\r'
              echo -e 'Access-Control-Allow-Headers: DNT, X-CustomHeader, Keep-Alive, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Authorization \r'
              echo -e '\r'

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"               echo -e "QUERY_STRING: ${QUERY_STRING}\r"
              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"

              exit 0
            • #!/bin/bash
              # when called with:
              #  curl -i "http://localhost/?a=b&c=d" --data "toto=tu" --data "toto=ta"
              # result is:
              #  REQUEST_METHOD: POST
              #  QUERY_STRING: a=b&c=d
              #  CONTENT_TYPE: application/x-www-form-urlencoded
              #  CONTENT_LENGTH: 15
              #  CONTENT: toto=tu&toto=ta

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"
              echo -e "QUERY_STRING: ${QUERY_STRING}\r"

              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"
              read -N ${CONTENT_LENGTH} CONTENT
              echo -e "CONTENT: ${CONTENT}\r"

              exit 0
            • #!/bin/bash -x
              echo -e "Content-type: text/plain\n"
              echo "content type: ${CONTENT_TYPE}"
              echo "QUERY_STRING_POST:"
              cat
            • #!/bin/bash -x
              echo -e "Content-type: text/plain\n"
              echo "content type: ${CONTENT_TYPE}"
              read -N $CONTENT_LENGTH QUERY_STRING_POST
              echo "QUERY_STRING_POST:"
              echo "$QUERY_STRING_POST"
          • parse_multi.sh
            • #!/bin/bash
              echo -e "Content-type: text/plain\n"

              echo "CONTENT_TYPE: $CONTENT_TYPE"
              echo "CONTENT_LENGTH: $CONTENT_LENGTH"

              # get boundary from CONTENT_TYPE
              boundary=$(echo $CONTENT_TYPE | awk -F= '{print$2}')
              echo "== boundary:"
              echo "$boundary"

              # read standard input
              read -N $CONTENT_LENGTH QUERY_STRING_POST
              echo "== received standard input:"
              echo "$QUERY_STRING_POST"

              # parse standard input in multipart/form-data
              post_params=$(echo "$QUERY_STRING_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS="&"} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print $1"="$3}')
              echo "== post_params:"
              echo "$post_params"

              exit 0
      • Perl
      • Python
    • WSGI (Python Web Server Gateway Interface)
    • ASGI (Asynchronous Server Gateway Interface)
  • Servlets / JSP servers
  • C Server Pages (CSP)
  • MIME types configuration

Servidors FTP / FTP servers
  • Clients FTP
  • vsftpd
  • ProFTPD
    • Instal·lació / Installation
      • CentOS
        • sudo yum install proftpd
        • systemctl start proftpd
    • Mini-Howto
      • Debugging
        • /usr/lib/systemd/system/proftpd.service
          • ExecStart = /usr/sbin/proftpd -d10 $PROFTPD_OPTIONS
      • Configuring a directory
      • Virtual users
        • Passos / Steps
          • sudo -i
          • cd /etc; mkdir proftpd; cd proftpd
          • wget https://raw.githubusercontent.com/proftpd/proftpd/master/contrib/ftpasswd
          • chmod +x ftpasswd
          • ./ftpasswd --group -gid 9000 --name ftp_users
          • ./ftpasswd --passwd --name user1 --home /home --shell /bin/bash --uid 8000 --gid 9000
          • /etc/proftpd.conf
            • # users
              AuthUserFile /etc/proftpd/ftpd.passwd
              AuthGroupFile /etc/proftpd/ftpd.group
          • systemctl restart proftpd.service
        • Test
          • ftp://user1...
      • AWS and ProFTPD
        • security groups
          • tcp 21
          • tcp 1024 - 1048
        • proftpd.conf
          • # aws
            PassivePorts        1024 1048
            MasqueradeAddress   <your_public_ip_address>
    • /etc/proftp.conf
      • DefaultRoot ~
      • <Directory ~>
            <Limit ALL>
              DenyAll
            </Limit>
            <Limit DIRS READ>
              AllowAll
            </Limit>
        </Directory>
        <Directory ~/upload>
            <Limit WRITE>
              AllowAll
            </Limit>
        </Directory>

Clients FTP / FTP Clients
  • sftp
  • lftp
    • download
      • lftpget ftp://user@remote_host/dir1/dir2/dir3/file1
      • lftp ftp://user@remote_host/dir1 -e "get dir2/dir3/file1; bye"
      • mirror
        • lftp ftp://user@remote_host/dir1 -e "mirror; bye"
    • upload
      • reverse (-R) mirror
        • lftp -d -f commands.lftp 2>> toto.log
          • commands.lftp
            • open -u user,password remote_host
              set ftp:ssl-allow no
              mirror -R local_dir
      • Exclude .git dir:
        • lftp exclude syntax confusion
        • commands.lftp
          • open -u user,password remote_host
            set ftp:ssl-allow no
            mirror --exclude --exclude=^\.git/$ -R local_dir remote_dir
      • How to use rsync over ftp
  • curlftpfs (mount locally)
  • weex (updating web pages)
  • Python

PHP
  • Pàgines web dinàmiques / Dynamic web pages
    • Frameworks
  • PEAR packages
  • PHP CLI (command line)
    • PHP CLI
    • Using PHP from the command line
    • Instal·lació / Installation
      • Mageia
        • urpmi php-cli
    • Exemples / Examples: (-r to avoid <?php...?>)
      • php [-f] toto.php
      • php -r "mail('to_user@example.org','subjecte','cos del missatge','From:from_user@example.org' );"
  • IDE
  • PHP debug
    • php -i
    • /etc/php.ini (Mageia: urpmi php-ini) or /etc/php5/apache2/php.ini (Debian/Ubuntu)
      • display_errors = On
    • service httpd restart (or: sudo service apache2 restart)
    • comprova la sintaxi / check syntax
      • urpmi php-cli
      • php -l toto.php
    • /etc/php.ini or /etc/php5/apache2/php.ini
    • logs
      • error_log
      • Examples
        • error_log("You messed up!", 3, "/var/tmp/my-errors.log");
        • error_log(print_r($variable, TRUE));
        • error_log("You messed up!", 3, "php://stdout");
    • see also Apache logs
    • check the output
      • php -f toto.php
      • if result is empty and "echo $?" returns 255, it could mean that dependencies not satisfied
    • xdebug
      • Instal·lació / Installation
        • Mageia
          • urpmi php-xdebug
        • Ubuntu
          • sudo apt-get install php5-xdebug
      • Eclipse
        • Debugging using XDebug
        • Setup php ini file:
          • Mageia: /etc/php.d/A29_xdebug.ini
          • Ubuntu: /etc/php5/apache2/conf.d/20-xdebug.ini
          • xdebug.remote_enable = 1
            xdebug.remote_handler = 'dbgp'
        • An http server must be configured and running
        • Eclipse: Window / Preferences / PHP / Servers
          • Server
            • Base URL: http://localhost
            • Document Root: /var/www/html
          • Debugger: XDebug
            • Port: 9000
        • Problems
          • Progress tab: "waiting for xdebug session"
            • Solution
              • xdebug.remote_handler = 'dbgp'
      • kdevelop
    • Debugging remote CLI with phpstorm
  • Dependencies
    • inclued
      • Installation
        • Mageia
          • urpmi php-inclued
  • Errors
    • Reference - What does this error mean in PHP?
    • "Warning: cannot modify header information..."
      • Si a php.ini hi ha:
        • error_reporting = E_ALL & ~E_DEPRECATED
        • i es produeix un "Notice:...", es posarà al text de resposta, i el codi serà un 200, i ja no es podrà  canviar per un altre
      • En canvi, si a php.ini hi ha:
        • error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE
        • no s'escriurà cap "Notice:..." a la resposta, i el codi HTTP es podrà canviar mitjançant un "header("HTTP/1.0 204 No Response")"
      • Penseu també  a eliminar les línies en blanc o retorns de carro fora de <php>
  • i18n
  • Smarty
    • i18n
      • smarty-gettext
        • Smarty gettext plugin
        • Installation
          • cd <smarty>/plugins
          • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/block.t.php
          • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/function.locale.php
        • Setup
          • cd public
          • get script to extract translatable strings
            • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/tsmarty2c.php
            • chmod +x tsmarty2c.php
          • edit your master tpl file:
            • {locale path="Locale" domain="messages"}
          • edit your_view.tpl
            • {t}Hello world from Smarty!{/t}
          • extract all translatable strings to a pot file:
            • ./tsmarty2c.php -o smarty.pot <path_to>/views/
          • merge pot files from *.php and *.tpl
            • xgettext --from-code=UTF-8 -o php.pot *.php
            • msgcat -o messages.pot php.pot smarty.pot
          • create po files from pot
            • cp messages.pot Locale/ca/LC_MESSAGES/messages.po
          • cd Locale/ca/LC_MESSAGES
          • edit messages.po to update translations
          • compile po to mo:
            • msgfmt -o messages.mo messages.po
        • Next times:
          • regenerate pot
            • xgettext --from-code=UTF-8 -o php.pot *.php
            • ./tsmarty2c.php -o smarty.pot <path_to>/views/
            • msgcat -o messages.pot php.pot smarty.pot
          • update po
            • msgmerge -U Locale/ca/LC_MESSAGES/messages.po messages.pot
            • msgmerge -U Locale/fr/LC_MESSAGES/messages.po messages.pot
            • ...
          • recompile mo
            • msgfmt -o Locale/ca/LC_MESSAGES/messages.mo Locale/ca/LC_MESSAGES/messages.po
            • msgfmt -o Locale/fr/LC_MESSAGES/messages.mo Locale/fr/LC_MESSAGES/messages.po
            • ...
  • Correu electrònic / E-mail
  • curl-php
    • PHP Curl with port number issue
    • POST of array to djangorestframework, in multipart (PhpMultiPartParser)
      • <?php
        // http://stackoverflow.com/questions/3772096/posting-multidimensional-array-with-php-and-curl
        function http_build_query_for_curl( $arrays, &$new = array(), $prefix = null ) {

            if ( is_object( $arrays ) ) {
                $arrays = get_object_vars( $arrays );
            }

            foreach ( $arrays AS $key => $value ) {
                $k = isset( $prefix ) ? $prefix . '[' . $key . ']' : $key;
                if ( is_array( $value ) OR is_object( $value )  ) {
                    http_build_query_for_curl( $value, $new, $k );
                } else {
                    $new[$k] = $value;
                }
            }
        }

        $ch = curl_init();
        $url = "http://127.0.0.1:8000/mymodels/";
        $myvalues = array("first", "second");
        $post_fields = array("name" => "toto", "myvalues" => $myvalues );

        http_build_query_for_curl( $post_fields, $post );

        print_r($post);

        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: multipart/form-data'));
        curl_setopt($ch, CURLOPT_POST, sizeof($post_fields));
        curl_setopt($ch, CURLOPT_POSTFIELDS, $post );

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        $server_output = curl_exec($ch);

        $info    = curl_getinfo($ch);
        if(curl_errno($ch)) {   
                    echo "something was wrong in your request";
                    }
        curl_close($ch);
        print $server_output;

        ?>
  • Certificats / Certificates
  • Cross-Origin Resource Sharing (CORS)
    • Requests with credentials
    • Server-side Acess Control (PHP examples)
    • http://foo.example/ (Javascript)
      • var invocation = new XMLHttpRequest();
        var url = 'https://bar.other/'

        invocation.onreadystatechange = function() {
          if (invocation.readyState!=4) return;

          var is_ok = invocation.responseText=='OK';
          if (is_ok) {
            document.location.href = isok.html
          } else {
                    document.location.href = isnotok.html
          }
        }

        invocation.open('GET', url, true );         invocation.withCredentials = "true";
        invocation.send();
    • https://bar.other/ (PHP)
      • <?php
        header('Access-Control-Allow-Origin: http://        foo.example/');
        header('Access-Control-Allow-Credentials: true');

        if ($_SERVER['SSL_CLIENT_VERIFY']=='SUCCESS') {
          echo 'OK';
        } else {           $sslcn = $_SERVER['SSL_CLIENT_S_DN_CN'];   $msg = "$sslcn NOT_OK";
          echo         $msg;
        }
        ?>
    • Nota: si no es fan servir Access-Control-Allow- (o bé "Access-Control-Allow-Origin: *"), no estarà disponible responseText a la URL origen. (Firefox: Cross-domain requests with credentials return empty)
    • "Access-Control-Allow-Origin: *" no pot estar lligat a withCredentials = "true"
  • HTML parsing
  • PHP3 (es) (ch)
  • Databases interaction
  • Emacs modes
    • Php Mode (EmacsWiki)
      1. cd /usr/share/emacs/24.2/lisp/progmodes/
      2. wget http://php-mode.svn.sourceforge.net/svnroot/php-mode/tags/php-mode-1.5.0/php-mode.el
      3. ~/.emacs
        • (autoload 'php-mode "php-mode" "Major mode for editing php code." t)
          (add-to-list 'auto-mode-alist '("\\.php$" . php-mode))
          (add-to-list 'auto-mode-alist '("\\.inc$" . php-mode))
  • LBD/EPFL
  • Manual (local)
  • Tutorial
  • PHP Nuke
  • Suhosin (Hardened-PHP project - PHP security)

Crawling

Recerca / Search

  • Search engine optimization (SEO) (wp)

Robots

Seguretat / Security

Servidors de correu / E-mail servers

Clients de correu electrònic / E-mail clients
  • Obrir un adjunt winmail.dat / Open a winmail.dat attachment:
    • Install tnef:
      • urpmi tnef
      • apt-get install tnef
    • Unpack attachement content:
      • tnef winmail.dat
  • Command line
    • How do I test an imap server?
    • Testing POP3 and IMAP servers from the command line in CMD or bash
    • imap:
      • telnet mail.example.com 143
      • ...
    • telnet/nc to an SMTP server (after EHLO, responses starting with 250 show available commands) (cannot continue a connection when STARTTLS has been invoked; use openssl s_client or mailx instead)
      • port 25 (default)
        • HELO (basic commands)
          • $ telnet mail.example.org 25
            Trying x.x.x.x...
            Connected to mail.example.org.
            Escape character is '^]'.
            220 client.example.org ESMTP Postfix
            HELO mail.example.org
            250             client.example.org
            MAIL FROM:<localuser@example.org>
            250 2.1.0 Ok
            RCPT TO:<remoteuser@example.org>
            250 2.1.5 Ok
            DATA
            354 End data with <CR><LF>.<CR><LF>
            To:<remoteuser@example.org>
            From:<localuser@example.org>
            Subject:First test
            Hi, you!
            .
            250 2.0.0 Ok: queued as 489C411EE7AC
            QUIT
            221 2.0.0 Bye
            Connection closed by foreign host.
        • EHLO (enhanced commands)
          • $ telnet mail.example.org 25
            Trying x.x.x.x...
            Connected to mail.example.org.
            Escape character is '^]'.
            220 client.example.org ESMTP Postfix
            EHLO mail.example.org 250-client.example.org
            250-PIPELINING
            250-SIZE 10240000
            250-VRFY
            250-ETRN
            250-STARTTLS
            250-ENHANCEDSTATUSCODES
            250-8BITMIME
            250 DSN
            ...
      • port 587 (must be activated; see Postfix on port 587) and STARTTLS:
        • $ telnet mail.example.org 587 Trying x.x.x.x...
          Connected to mail.example.org.
          Escape character is '^]'.
          220 client.example.org ESMTP Postfix
          EHLO mail.example.org 250-client.example.org
          250-PIPELINING
          250-SIZE 10240000
          250-VRFY
          250-ETRN
          250-STARTTLS
          250-ENHANCEDSTATUSCODES
          250-8BITMIME
          250 DSN
          STARTTLS
          ...
    • openssl s_client
      • openssl s_client -connect mail.example.org:587 -starttls smtp
      • openssl s_client -starttls smtp -connect smtp.gmail.com:587 -crlf -ign_eof
      • Problemes / Problems
        • read:errno=0 (and connection is closed):
          • check, on postfix server: /var/log/maillog
    • mail / mailx
      • Instal·lació / Installation
        • CentOS
          • sudo yum install mailx
      • Utilització / Usage
  • Mozilla Thunderbird (correu/mail)
    • Command line options
      • create a new profile
        • thunderbird -CreateProfile my_profile
        • it will be created at:
          • ~/.thunderbird/xxxxxxxx.my_profile/
      • open gui profile manager
        • thunderbird -P
      • open a profile
        • thunderbird -p my_profile
    • Thunderbird 3
    • Redacció d'un missatge en HTML quan per omissió és en text pla / Compose a message in HTML when default is plain text:
      • Majúscula + click a "Redacta" / Shift + click on "Compose"
    • Expanded columns in folder pane
    • Signatures - Thunderbird
    • enllaços cap a firefox i altres aplicacions: ~/.thunderbird/.../prefs.js:
      • Opening hyperlinks from some GTK-based applications doesn't work after updating Firefox to a new version
      • Bug 58784 - Using "Make firefox the default web browser" should use /usr/bin/firefox not /usr/lib(64)/firefox-<version>/firefox
      • $ gconf-editor
        • /desktop/gnome/url-handlers/http/command
          • enabled: no
          • com que el valor per omissió és "enabled=true", això crearà el fitxer .gconf/desktop/gnome/url-handlers/http/%gconf.xml (si no, no existeix el fitxer)
        • llavors el thunderbird ens preguntarà amb quina aplicació volem obrir els http: /usr/bin/firefox
      • gconf: ~/.gconf/desktop/gnome/url-handlers/http/%gconf.xml
      • Al Firefox: Edita / Preferències / Avançat / General / Comprova-ho ara (predeterminat) (*)
      • user_pref("network.protocol-handler.app.http", "/usr/bin/xdg-open");
      • user_pref("network.protocol-handler.app.http", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.https", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.ftp", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.smb", "/usr/bin/konqueror");
    • Locale-Switcher Extension 
    • Tips & tricks
    • Consells i trucs (Softcatalà)
    • Use different Quote Level Colors
      • ~/.thunderbird/xxx.default/chrome/userContent.css
        • /* Quote Levels Colors */
          blockquote[type=cite] {
             color: navy !important; background-color: RGB(245,245,245) !important;
          }
          blockquote[type=cite] blockquote {
             color: maroon !important; background-color: RGB(235,235,235) !important;
          }
          blockquote[type=cite] blockquote blockquote {
             color: green !important; background-color: RGB(225,225,225) !important;
          }
          blockquote[type=cite] blockquote blockquote blockquote {
             color: purple !important; background-color: RGB(215,215,215) !important;
          }
          blockquote[type=cite] blockquote blockquote blockquote blockquote {
             color: teal !important; background-color: RGB(205,205,205) !important;
          }
    • Problem: "This body part will be downloaded on demand"
      • View->Display Attachments Inline
    • how to not include external images on html messages? - Mozilla
      • moz-do-not-send true
      • En recepció, caldrà acceptar "Mostra el contingut remot"
    • Lectura de fitxers mbox / Read mbox files
      • Reading an mbox file with Thunderbird
      • min_thunderbird.sh
        • #!/bin/bash

          EXPECTED_ARGS=1
          if (( $# != $EXPECTED_ARGS ))
          then
              cat <<EOF
          Usage: `basename $0` profile_name

          Create a thunderbird profile with given name, and add a default mail setup, in order to read mbox files.
          EOF
              exit 1
          fi

          # check that thunderbird is installed
          if ! command -v thunderbird2 >/dev/null 2>&1
          then
              echo "ERROR: command thunderbird does not exist. Please install it (https://www.thunderbird.net/) before running this script."
              exit 1
          fi

          profile_name=$1

          root_thunderbird_dir=${HOME}/.thunderbird

          # create profile
          echo "1. creating profile: ${profile_name}"
          thunderbird -CreateProfile ${profile_name}

          # get profile path
          profile_dirname=$(find ${HOME}/.thunderbird -type d -name "*\.${profile_name}")
          echo "  created: ${profile_dirname}"

          # start thunderbird, to create dirs
          echo "2. starting thunderbird to create dirs: please just cancel and close"
          thunderbird -p ${profile_name}


          # backup orginal prefs.js
          prefs_path=${profile_dirname}/prefs.js
          original_prefs_path=${profile_dirname}/prefs.original.js
          echo "3. backing up ${prefs_path} -> ${original_prefs_path}"
          cp ${prefs_path} ${original_prefs_path}

          # modify prefs.js
          echo "4. modifying ${prefs_path} to add a minimal mail config"
          cat >>${prefs_path} <<EOF
          user_pref("mail.root.none", "${profile_dirname}/Mail");
          user_pref("mail.accountmanager.accounts", "account1");
          user_pref("mail.accountmanager.localfoldersserver", "server1");
          user_pref("mail.account.account1.server", "server1");
          user_pref("mail.server.server1.directory", "${profile_dirname}/Mail/Local Folders");
          user_pref("mail.server.server1.directory-rel", "[ProfD]Mail/Local Folders");
          user_pref("mail.server.server1.hostname", "Local Folders");
          user_pref("mail.server.server1.name", "Carpetes locals");
          user_pref("mail.server.server1.nextFilterTime", 10);
          user_pref("mail.server.server1.spamActionTargetAccount", "mailbox://nobody@Local%20Folders");
          user_pref("mail.server.server1.storeContractID", "@mozilla.org/msgstore/berkeleystore;1");
          user_pref("mail.server.server1.type", "none");
          user_pref("mail.server.server1.userName", "nobody");
          EOF

          # start thunderbird
          echo "5. starting thunderbird: just check that default mail folders have been created and close it"
          thunderbird -p ${profile_name}

          # additional mbox files
          echo "6. to read mbox files, put them on (or link them from) ${profile_dirname}/Mail/Local Folders/ and restart thunderbird by using one of the following:"
          echo "  thunderbird -p ${profile_name}"
          echo "  thunderbird -P"
          echo "To delete profile ${profile_name}:"
          echo "  thunderbird -P"
          echo "and select Delete Profile..."
          echo "IMPORTANT: if you made a symbolic link to your mbox file from ${profile_dirname}/Mail/Local Folders/, you can choose Delete Files option. Otherwise, choose Don't Delete Files."

          exit 0
  • Gmail

Navegadors / Browsers

HTML

CSS

XHTML

XML

  • Extensible Markup Language (XML) (W3C)
  • Schemas
  • The XML library for Gnome
  • <recursos XML/>
  • XML Pitstop
  • Search
    • XPath
      • XPath tutorial (w3 schools)
      • Free Online XPath Tester / Evaluator
      • ...

        		 <root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org">
          <actors>
            <actor id="1">Christian Bale</actor>
            <actor id="2">Liam Neeson</actor>
            <actor id="3">Michael Caine</actor>
          </actors>
          <foo:singers>
            <foo:singer id="4">Tom Waits</foo:singer>
            <foo:singer id="5">B.B. King</foo:singer>
            <foo:singer id="6">Ray Charles</foo:singer>
          </foo:singers>
        </root>
        selecciona el node del document /
        selecciona el node 'root' /root
        selecciona tots els elements 'actor' que són directament fills de l'elelement 'actors' /root/actors/actor
        selecciona tots els elements 'singer', siguin on siguin al document //foo:singer
        selecciona l'atribut 'id' dels elements 'singer', siguin on siguin //foo:singer/@id
        selecciona el valor del text del primer element 'actor' //actor[1]/text()
        selecciona l'últim element 'actor' //actor[last()]
        selecciona el primer i segon elements 'actor' segons la seva posició //actor[position() < 3]
        selecciona tots els elements 'actor' que tinguin atribut 'id' //actor[@id]
        selecciona l'element 'actor' que té l'atribut 'id' amb valor '3' //actor[@id='3']
        selecciona l'element 'actor' que té l'atribut 'id' amb valor més petit o igual que '3' //actor[@id<='3']
        selecciona totsel fills del node 'singers' /root/foo:singers/*
        selecciona tots els elements del document //*
        selecciona els elements 'actor' i els elements 'singer' //actor|//foo:singer
        selecciona el nom del primer element del document name(//**[1])
        selecciona el valor numèric de l'atribut del primer element 'actor' number(//actor[1]/@id)
        selecciona la representació en text del valor de l'atribut 'id' del primer element 'actor' string(//actor[1]/@id)
        selecciona la longitud del valor textual del primer element 'actor' string-length(//actor[1]/text())
        selecciona el nom local del primer element 'singer' (és a dir, sense espais) local-name(//foo:singer[1])
        selecciona el nombre d'elements 'singer' count(//foo:singer)
        selecciona la suma dels atributs 'id' dels elements 'singer' sum(//foo:singer/@id)
  • Validate
  • Style
    • XSL (EXtensible Stylesheet Language)
      • style sheets for XML (CSS = style sheets for HTML)
  • Transform
    • Formats
      • XSLT
        • language desc tutorial reference functions
          XSLT (XSL Transformations) transforming XML documents XSLT Introduction XSLT elements reference
          		 Functions reference
          XPath navigating in XML documents XPath tutorial

          XQuery querying XML documents


        • XML and XSLT (w3schools)
        • Conversion XML -> HTML
          • browser will take file specified on xml-stylesheet to render XML as HTML; if no xml-stylesheet is specified, it will render raw xml
          • cdcatalog.xml cdcatalog.xsl (xml -> html) cdcatalog_text.xsl (xml -> txt)
            <?xml version="1.0" encoding="UTF-8"?>
            <?xml-stylesheet type="text/xsl" href="cdcatalog.xsl"?>
            <catalog>
              <cd>
                <title>Empire Burlesque</title>
                <artist>Bob Dylan</artist>
                <country>USA</country>
                <company>Columbia</company>
                <price>10.90</price>
                <year>1985</year>
              </cd>
            .
            .
            </catalog>             		<?xml version="1.0" encoding="UTF-8"?>

            <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
            <xsl:output method="html"/>
            <xsl:template match="/">
              <html>
              <body>
              <h2>My CD Collection</h2>
              <table border="1">
                <tr bgcolor="#9acd32">
                  <th>Title</th>
                  <th>Artist</th>
                </tr>
                <xsl:for-each select="catalog/cd">
                              <xsl:sort select="artist"/>       <tr>
                    <td><xsl:value-of select="title"/></td>
                    <td><xsl:value-of select="artist"/></td>
                  </tr>
                </xsl:for-each>
              </table>
              </body>
              </html>
            </xsl:template>

            </xsl:stylesheet>                         		<?xml version="1.0" encoding="UTF-8"?>

            <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
              <xsl:output method="text" media-type="text/plain"/>

              <xsl:template match="/">
                <xsl:for-each select="catalog/cd">
                  <xsl:value-of select="title"/>
                  <xsl:text>: </xsl:text>
                  <xsl:value-of select="artist"/>
                  <xsl:text>&#xa;</xsl:text>
                </xsl:for-each>
              </xsl:template>
             
            </xsl:stylesheet>
        • Conversion XML -> JSON
    • Tools
    • Exemples / Examples
  • Editors 
  • Biblioteques / Libraries
  • Compressió / Compression
  • Eines / Tools
    • Command-line XML processing
    • How to parse XML using shellscript? [duplicate]
    • xmlstarlet (transform, query, validate and edit)
      • Documentation (single html)
      • Instal·lació / Installation
        • Mageia
          • urpmi xmlstarlet
          • binary is xmlstartlet instead of xml
      • Changing the value of an XML element depending on the value of another element in the tree
      • Searching for XML tag by value between them and inserting a new tag in shell script
      • Estructura / Structure:
        • xmlstarlet el toto.xml
      • Reformata / Reformat (prettify)
        • xmlstarlet format --encode utf-8 input.xml >output.xml
      • Validating XML documents (using XSD, DTD)
        • xml val -s toto.xsd -e toto.xml >validation_output.txt 2>&1
        • xmlstarlet val -s toto.xsd -e toto.xml >validation_output.txt 2>&1
      • Transforming XML documents (using XSLT)
        • xml tr xsl/param1.xsl xml/table.xml
        • xmlstarlet tr xsl/param1.xsl xml/table.xml
        • set parameters defined in param1.xsl as <xsl:param name="my_param">
          • xmlstarlet tr xsl/param1.xsl -s my_param=my_value xml/table.xml
      • Print to plain text
        • How to convert multi level xml to a single line containing all levels using XMLSTARLET
        • XMLStarlet: Printing one line per item, while using datum from parent element
        • Use XMLStarlet to parse XML in the Linux terminal
        • xmlstarlet sel ...
          • -C - show xsl generated code
          • -T - text output
          • -t - next arguments specify a template
            • -m, --match <xpath> - match
            • -v, --value-of <xpath> - print value (between open and closing tags)
            • -o <string> - print literal string
            • -n - print newline
            • ...
        • Exemples / Examples
          • myfile.xml
            • <?xml version="1.0" encoding="UTF-8" standalone="no"?>
              <xml>
                <os>
                 <linux>
                  <distribution when="today">
                    <name>Fedora</name>
                    <release>7</release>
                    <codename>Moonshine</codename>
                    <clau usage="signing">
                      clau per signar
                    </clau>
                    <clau usage="encryption">
                      clau per xifrar
                    </clau>
                    <spins>
                      <name>Live</name>
                      <name>
                        Fedora
                      </name>
                      <name>Everything</name>   
                    </spins>
                  </distribution>

                  <distribution when="tomorrow">
                    <name>Fedora Core</name>
                    <release>6</release>
                    <codename>Zod</codename>
                    <spins></spins>
                  </distribution>
                 </linux>
                </os>   
              </xml>
          • estructura:
            • xmlstarlet el input.xml
            • sense ordre ni repetició:
              • xmlstarlet el -u input.xml
            • amb atributs:
              • xmlstarlet el -a input.xml
            • amb els valors dels atributs:
              • xmlstarlet el -v input.xml
          • parts a partir de distribution, sense les etiquetes, substituïdes per espais buits:
            • xmlstarlet select --template --value-of /xml/os/linux/distribution input.xml
          • si voleu tot l'xml:
            • xmlstarlet select --template --copy-of /xml/os/linux/distribution input.xml
          • només el contingut de distribution/name:
            • xmlstarlet select --template --value-of /xml/os/linux/distribution/name --nl myfile.xml
          • selecció condicional, fent servir XPath:
            • xmlstarlet sel --template --value-of '/xml/os/linux/distribution[name = "Fedora Core"]/release' --nl myfile.xml
            • xmlstarlet sel --template --value-of '/xml/os/linux/distribution[name = "Fedora Core"]/codename' --nl myfile.xml
            • filtre per atribut, amb sortida xml (--copy-of):
              • xmlstarlet sel --template --match '/xml/os/linux/distribution/clau[@usage = "encryption"]' --copy-of '.' --nl myfile.xml
          • match seguit de value-of:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution/spins' --value-of '.' --nl myfile.xml
            • xmlstarlet sel --template --match '/xml/os/linux/distribution' --value-of 'spins' --nl myfile.xml
          • atribut:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution' --value-of '@when' --nl myfile.xml
          • si voleu eliminar els espais i canvis de línia:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution/spins/name' --value-of 'normalize-space()' --nl myfile.xml
            • xmlstarlet sel --noblanks --template --match '/xml/os/linux/distribution/clau[@usage = "encryption"]' --value-of 'normalize-space()' myfile.xml
          • si el fitxer especifica xmlns (namespace):
            • myfile_ns.xml
              • <?xml version="1.0" encoding="UTF-8" standalone="no"?>
                <xml xmlns="...">
                  <os>
                   <linux>
                    <distribution when="today">
                      <name>Fedora</name>
                      <release>7</release>
                      <codename>Moonshine</codename>
                      <spins>
                        <name>Live</name>
                        <name>Fedora</name>
                        <name>Everything</name>   
                      </spins>
                    </distribution>

                    <distribution when="tomorrow">
                      <name>Fedora Core</name>
                      <release>6</release>
                      <codename>Zod</codename>
                      <spins></spins>
                    </distribution>
                   </linux>
                  </os>   
                </xml>
            • xmlstarlet sel --template --match '//_:xml' --value-of '_:os' --nl myfile_ns.xml
            • xmlstarlet sel --template --match '//_:xml' --value-of '_:os/_:linux/_:distribution/_:name' --nl myfile_ns.xml
            • SAML
              • xmlstarlet sel --template --match '//_:EntityDescriptor/_:AttributeAuthorityDescriptor/_:KeyDescriptor[@use="signing"][1]/ds:KeyInfo/ds:X509Data/ds:X509Certificate' --value-of 'normalize-space()' metadata.xml
              • echo -e "-----BEGIN CERTIFICATE-----$(echo "$(xmlstarlet sel --template --match '//_:EntityDescriptor/_:AttributeAuthorityDescriptor/_:KeyDescriptor[@use="signing"][2]/ds:KeyInfo/ds:X509Data/ds:X509Certificate' --value-of '.' metadata.xml)" | tr -d ' ')\n-----END CERTIFICATE-----" >cert_5.txt
              • (no acaba de funcionar) xmlstarlet sel --template --var nl -n -b --match '//_:EntityDescriptor/_:AttributeAuthorityDescriptor/_:KeyDescriptor[@use="signing"][2]/ds:KeyInfo/ds:X509Data' -o "-----BEGIN CERTIFICATE-----" --value-of 'concat(ds:X509Certificate,$nl,"-----END CERTIFICATE-----")' --nl metadata.xml
          • ...
        • Additional functions from exslt (can be used inside -v):
        • print to csv
          • info
          • passos / steps
            1. get xsl code (-C) by running: xmlstarlet sel -C ... >my_xml_to_csv.xsl
            2. edit the bottom part of my_xml_to_csv.xsl, to replace template value-of-template:
              • <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exslt="http://exslt.org/common" version="1.0" extension-element-prefixes="exslt">

                  ...
                 
                  <xsl:template name="value-of-template-old">
                    <xsl:param name="select"/>
                    <xsl:value-of select="$select"/>
                    <xsl:for-each select="exslt:node-set($select)[position()&gt;1]">
                      <xsl:value-of select="'&#10;'"/>
                      <xsl:value-of select="."/>
                    </xsl:for-each>
                  </xsl:template>

                 
                  <xsl:variable name="quot">"</xsl:variable>
                  <xsl:variable name="doublequot">""</xsl:variable>
                 
                  <xsl:template name="value-of-template">
                    <xsl:param name="select" />
                    <xsl:choose>
                      <xsl:when test="contains($select, $quot)">
                        <xsl:value-of select="substring-before($select,$quot)" />
                        <xsl:value-of select="$doublequot" />
                        <xsl:call-template name="value-of-template">
                          <xsl:with-param name="select" select="substring-after($select,$quot)" />
                        </xsl:call-template>
                      </xsl:when>
                      <xsl:otherwise>
                        <xsl:value-of select="$select" />
                      </xsl:otherwise>
                    </xsl:choose>
                  </xsl:template>

                </xsl:stylesheet>
              • ...
            3. run xmlstarlet tr with the xsl file:
              • xmlstarlet tr my_xml_to_csv.xsl toto.xml >toto.csv
            4. you may want to convert from csv to xlsx
        • ...
      • ...
    • xmlindent
    • Xalan command line utility (Apache)
    • Sun Multi-Schema Validator (MSV)
      • rngconv: conversion (.xsd,.dtd) -> .rng (RelaxNG XML syntax)
        • java -jar rngconv.jar toto.xsd > toto.rng
        • java -jar rngconv.jar -dtd toto.dtd > toto.rng
    • Trang (RelaxNG conversion: .rng (XML syntax) -> .rnc(compact syntax))
      • java -jar trang.jar toto.rng toto.rnc
  • Trucs / Tips
  • Diferències / Differences (text diff)

JavaScript

  • ECMAScript
  • Documentation
  • Equivalències

    		 JavaScript Python
    Data structures
    • undefined: an identifier, absence of a value
    • null: a keyword, absence of an object
    Objects Definition of an object:
    const obj = {
      name: "Carrot",
      for: "Max",
      details: {
        color: "orange",
        size: 12,
      },
    };     Access to its properties:
    • obj.name
    • obj["name"]
    		Definition of a dictionary:
    obj = {
      "name": "Carrot",
      "for": "Max",
      "details": {
        "color": "orange",
        "size": 12,
      }
    }     Access to its properties:
    • obj["name"]
    Arrays for (const i of mylist) for i in mylist
    Functions ...args
    		 *args

    		Basic function / Function declaration:
    function add(x, y) {
      const total = x + y;
      return total;
    }

    		Anonymous function / Function expression:
    const myvar = function (...args) {
      // ...
    }

    		Anonymous arrow function:
    const myvar = (...args) => {
      // ...
    }    		 Lambda:
    ...

    		Immediately invoked function expression (IIFE):
    (function () {
      // ...
    })();    		 Lambda IIEF:
    (lambda x, y: x + y)(2, 3)
    Classes Class declaration:
    class MyClass {
      constructor(...) {
        // ...
      #my_private_field ...
      static my_static_method ...
      }
    }

    // Declaration
    class Rectangle {
      constructor(height, width) {
        this.height = height;
        this.width = width;
      }
    }     		class MyClass:
      def __init__(self, ...):
        ...
      _my_private_method ...

    		Class expression:
    // Expression; the class is anonymous but assigned to a variable
    const Rectangle = class {
      constructor(height, width) {
        this.height = height;
        this.width = width;
      }
    };

    // Expression; the class has its own name
    const Rectangle = class Rectangle2 {
      constructor(height, width) {
        this.height = height;
        this.width = width;
      }
    };

    		Inheritance:
    class Dog extends Animal {
      ...
    }    		 class Dog(Animal):
    Asynchronous Callback-based:
    fs.readFile(filename, (err, content) => {
      // This callback is invoked when the file is read, which could be after a while
      if (err) {
        throw err;
      }
      console.log(content);
    });
    // Code here will be executed while the file is waiting to be read

    		Promise-based:
    fs.readFile(filename)
      .then((content) => {
        // What to do when the file is read
        console.log(content);
      })
      .catch((err) => {
        throw err;
      });
    // Code here will be executed while the file is waiting to be read

    		async/await:
    async function readFile(filename) {
      const content = await fs.readFile(filename);
      console.log(content);
    }
    Modules import { foo } from "./foo.js"; import foo
    from foo import bar
  • Tutorials
  • Reference
  • JavaScript Kit
  • JavaScript Gamelan Directory
  • DevEdge (Netscape)
  • Sash
  • JavaScript Calendar 
  • Free Cut-and-Paste JavaScript
  • Javascript PC emulator (micro Linux)
  • JSON
    • Comparison to XML
    • JavaScript Object Notation JSON
    • Search
    • JSON Schema
    • JSON-LD (Linking Data)
    • Parsers
      • Parsing json with sed and awk
      • How to parse JSON string via command line on Linux (jq)
      • Python
      • command line
        • jsawk
          • resty (script wrapper for curl)
        • jq
          • jq play (interactive player)
          • removal of nodes, by naming nodes to excise #37
            • del()
          • Ús / Usage
            • Tutorial
            • Manual
            • Cookbook
            • jq [options] '<commands>' file.json
            • jq -f file.jq ...
              • file.jq
                • <sequential_command_1> | <sequential_command_2> | (<parallel_command_3.1>), (<parallel_command_3.2>) | ...
            • cat file.json | jq [options] '<commands>'
            • echo $my_var | jq [options] '<commands>'
            • special character
              		 description
              |
              		 separate chained commands
              ()
              		 group commands
              ,
              		 separate parallel commands
              +
              		 concatenate strings
            • Jq to replace text directly on file (like sed -i)
              • jq '...' toto.json
          • Exemples / Examples
            • AWS
            • mostra el contingut del fitxer / show the file content
              • jq '.' toto.json | sponge toto.json
            • mostra el contingut de la variable / show the variable content
              • echo $my_var | jq '.'
            • multiple input files
            • mostra una línia per a cada entrada, amb dos dels camps separats per un espai:
              • jq -r '.[] | [.field_1, .field_2] | join(" ")'
            • posa cada element d'una llista en una sola línia / put each element of an array into a single line
              • toto.json
                • {
                    "llista": [
                      {
                        "nom": "primer",
                        "valor": 1,
                        "preu": 11
                      },
                      {
                        "nom": "segon",
                        "valor": 2,
                        "preu": 22
                      }
                    ]
                  }
              • jq -r '.llista[] | [.nom, .valor | tostring ] | join(" ")' toto.json
                • primer 1
                  segon 2
            • crea un array amb els valors d'un sol dels camps:
              • jq '[.llista[] | .nom]' toto.json
            • posa en una sola línia, un sol camp per a cada entrada:
              • jq '[.llista[] | .nom] | join(" ")' toto.json
              • ordenats:
                • jq '[.llista[] | .nom] | sort | join(" ")' toto.json
              • jq '.llista[] | .nom' toto.json | paste -sd' '
                • "primer" "segon"
              • es pot posar en un array de bash:
                • declare -a my_array
                • (?) my_array=($(echo $my_json | jq '.llista[] | []'jq '.llista[] | .nom' toto.json | paste -sd' '))
                • my_array=($(echo $my_json | jq '.llista[] | .nom' | paste -sd' '))
                • my_array=($(jq '.llista[] | .nom' toto.json | paste -sd' '))
            • posa en una sola línia la clau i el seu valor:
              • jq 'to_entries[] | [.key, .value] | join(" ")' toto.json
              • i es posa en un array associatiu de bash:
                • #!/bin/bash

                  input="{
                  \"el primer\": \"un u\",
                  \"el segon\": \"un dos\",
                  \"el tercer\": \"un tres\"
                  }"

                  declare -A my_array

                  while IFS= read -r element
                  do
                      echo "-- read element from jq: $element"
                      key=$(echo "$element" | awk 'BEGIN{FS="#"} {print $1}')
                      value=$(echo "$element" | awk 'BEGIN{FS="#"} {print $2}')
                      my_array["$key"]="$value"
                  done < <(echo "$input" | jq -r 'to_entries[] | [.key, .value] | join("#")')

                  for key in "${!my_array[@]}"
                  do
                      echo "${key}: ${my_array[$key]}"
                  done

                  exit 0
            • create json
              • jq -n '{foo:"bar",foo2:3}'
              • jq -n '.foo="bar"'
              • jq -n '.[0].foo=1'
            • variables bash
            • Afegeix entrades a una llista / Add entries to a list
              • part_1=$(jq -n '[{foo:"bar",foo2:3}]')
                part_2=$(jq -n '[{foo:"bar2",foo2:4}]')
                part_1=$(echo $part_1 | jq ". += $part_2")
                echo $part_1 | jq '.'
            • Substitució / Replacement
            • Manipulate json as a bash variable:
              • body=$(jq -n '{foo:"bar",foo2:3}')
                echo "${body}" | jq '.'

                body=$(echo "${body}" | jq '. += {foo3:"bar3"}')
                echo "${body}" | jq '.'

                value=bar4
                body=$(echo "${body}" | jq ". += {foo3:\"${value}\"}")
                echo "${body}" | jq '.'

                object=$(jq -cr -n '{foo4:"bar4",foo5:5}')
                echo "${object}" | jq '.'
                body=$(echo "${body}" | jq ". += {foo6:${object}}")
                echo "${body}" | jq '.'
            • Subconjunt / Subset
              • jq '{field1:.field1,field2:.field2}'
            • comprova si existeix my_key / check if my_key exists
              • if has("my_key") then .my_key else empty end
              • # desaconsellat, perquè si la clau existeix is és un booleà fals es complirà l'if i posarà empty
                my_value=$(cat toto.json | jq -r 'if .my_key then .my_key else empty end')
                if ! [ "$my_value" ]
                then
                    exit 1
                fi
            • if "my_key" exists, output "-my_key <my_key_value>"
              • (.my_key | values | tostring | "-my_key "+.),
            • fallback (alternative operator)
              • si el valor és null, retorna una cadena buida
                • my_value=$(echo ${curl_response} | jq '.my_key // empty')
              • si el valor és null, retorna una cadena emmagatzemada en una variable de la shell:
                • my_fallback_value="toto"
                  my_value=$(echo ${curl_response} | jq ".my_key // ${my_fallback_value}")
            • filters
              • @csv
              • @tsv
            • map
              • map(x) is the same as:
                • [.[] | x]
            • find / select
              • ".nginx.applications[] | select(.name ==\"$application_name\") | .record"
              • Només la primera / Only the first occurrence:
                • ".nginx.applications[] | map( select(.name ==\"$application_name\") | .record) | if .[0] then .[0] else empty end"
              • select elements with name="my_prefix...":
                • '.my_list[] | select(.name | startswith("my_prefix") )'
              • How to filter an array of objects based on values in an inner array with jq?
              • boolean
                • select(... and ...)
                • select(... or ...)
              • key and value
                • Select objects based on value of variable in object using jq
                • Filter objects based on tags in an array
                • given the following json:
                  • [
                        {
                            "title": "first",
                            "number": 1,
                            "tags": [
                                {"key": "foo1","value": "bar1"},
                                {"key": "foo2","value": "bar2"}
                            ]
                        },
                        {
                            "title": "second",
                            "number": 2,
                            "tags": [
                                {"key": "foo1","value": "bar2"},
                                {"key": "foo2","value": "bar1"}
                            ]
                        },
                        {
                            "title": "third",
                            "number": 3,
                            "tags": [
                                {"key": "foo1","value": "bar1"}
                            ]
                        }
                    ]
                  • select elements with a tag foo1/bar1 (first, third, but not second):
                    • jq '.[] | select( .tags[] | . and .key=="foo1" and .value=="bar1")'
              • remove empty elements
                • | select( length>0 )
            • Shellscript: loop through a JSON array with jq
              • while IFS= read -r element
                do
                  echo "-- read element from jq: $element"
                done < <(echo "$input" | jq -c '.toto[]')
              • while IFS= read -r; do
                  echo "read content from jq: $REPLY"
                done < <(echo "$input" | jq -c '.toto[]')
              • while IFS= read -r element
                do
                  echo "-- read element from jq: $element"
                done < <(jq -c '.toto[]' $filename)
            • sort (descending) by sum of several fields
              • [
                .[] | ([.components[] | .bandwidth] | add ) as $total_bandwidth |
                .+{tbw:$total_bandwidth}
                ] |
                sort_by(-.tbw)
            • sort and swap first and second elements of an array
              • 'sort_by(-.field_1) | .[1] as $second | .[1] = .[0] | .[0] = $second | .[]'
            • counter
            • aplana la jerarquia de claus / flatten key hierarchy
            • convert json to  ffmpeg parameters
              • .[] |
                [
                  ("-vsync vfr"),
                  (.components[] |
                    if .type!="image" then
                      (if .type=="video" then "v" else "a" end) as $tipus |
                      #.lang
                      "-c:"+$tipus, .codec,
                      (if .codec=="aac" then "-strict -2" else empty end),
                      (.preset | values | if (. | length) > 0 then "-preset "+. else empty end),
                      #(.preset | values | "-"+$tipus+"pre "+.),
                      "-b:"+$tipus, (.bandwidth | tostring),
                      (.channels | values | tostring | "-ac "+.),
                      (if .width then "-vf scale=w="+(.width | tostring)+":h="+(.height | tostring) else empty end),
                      #(if .width then "-vf scale=w="+(.width | tostring)+":h="+(.height | tostring)+":force_original_aspect_ratio=decrease" else empty end),
                      #(if .width then "-vf scale='-2:ceil(min(1\\,min("+(.width | tostring)+"/iw\\,"+(.height | tostring)+"/ih))*ih)'" else empty end),
                      (.gop | values | tostring | "-g "+.),
                      (.profile | values | if (. | length) > 0 then "-profile:"+$tipus+" "+. else empty end),
                      #(.profile | values | if (. | length) > 0 then "-profile "+. else empty end),
                      (.level | values | tostring | "-level "+.),
                      (.keyint_min | values | tostring | "-keyint_min "+.),
                      (.sc_threshold | values | tostring | "-sc_threshold "+.)
                    else
                      empty
                    end
                  ),
                  (.segment_duration | tostring | "-hls_time "+.),
                  (.segment_list_size | tostring | "-hls_list_size "+.),
                  (.segment_wrap | tostring | "-hls_wrap "+.),
                  (.segment_start_number | tostring | "-start_number "+.),
                  .name + ".m3u8"
                ]
                | join(" ")
    • Print
      • cat toto.json | python -m json.tool
    • Documentation
    • Conversion
      • Typson (TypeScript -> json-schema)
  • Debugging Javascript
  • JSONP (wp)
  • Toolkits, Frameworks
    • Comparison of Javascript frameworks
    • CAAT (multi-instance director-based scene-graph manager) (animacions 2D)
    • APE project (Ajax Push Engine)
    • mootools
    • Node.js
      • Getting started 

      		• name
        		 info
        		 installation
        		 usage
        Node.js
        		Installation of Node.js on system
        		 System:
        • Install NodeJS (Angular JS 1: Tutorial)
        • Mageia
          • urpmi nodejs (also installs npm)
        • CentOS
          • sudo yum install nodejs
        • Debian / Ubuntu
          • sudo apt-get install nodejs-legacy

        nvm Node.js version management:
        installation of several versions of Node.js on user home (~/.nvm/)
        • no need to install Node.js on system
        • npm install -g nvm (NVM in npm is not the right one? #304)
        • Installing and updating
          • curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
          • curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.39.1/install.sh | bash
        • (logout and login)
          • source ~/.bashrc
        • using Fabric
          • c.run("curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash")
            c.run("source ~/.nvm/nvm.sh && nvm install 20")
        • see Janus (WebRTC)
        • install and register the latest version of Node.js
          • nvm install node
        • install and register a specific version of Node.js
          • nvm install 14.7
        • set an alias for default
          • nvm alias default 14.7
        • switch to a registered version
          • nvm use default
        • switch to system version
          • nvm use system
        • show current version
          • nvm current
        • list installed versions
          • nvm ls
        • search for a specific version
          • nvm which 14.7
        Package manager
        		 npm
        • Mageia
          • urpmi npm (no needed if installed nodejs)
        • CentOS
          • sudo yum install npm
        • Debian / Ubuntu
          • sudo apt-get install npm
        • npm -g install npm@latest
        • from package.json, install locally to node_modules/
          • npm install
        • registered package (and install globally: -g, to /usr/lib/node_modules/ or ~/.nvm/versions/node/vx.y.z/lib/node_modules/)
          • npm install -g angular-cli
          • [sudo] npm install -g ...
        • install locally (no -g) in node_modules/ and register to package.json (--save):
          • npm install ... -save
        • list of locally (upwards recursivity) installed packages
          • npm list
        • list of globally installed packages
          • npm -g list
        Packages
        		 angular-cli
        		 AngularJS npm install -g angular-cli Used by Eclipse plugin: Angular2 Eclipse
        Bower packages for deployment manager
        • frameworks
        • libraries
        • assets
        • utilities

        		 npm install -g bower
        • from bower.json, install to your_project/app/bower_components/
          • bower install
        • registered package
        CoffeeScript little language that compiles into JavaScript npm install -g coffee-script
        Grunt Javascript task runner
        http-server
        		 HTTP server
        		 npm install -g http-server
      • npm
    • AngularJS (Google)
    • ember.js (wp)
    • jQuery (wp)
    • qooxdoo (wp)
  • Video players (HTML5)
  • Editors
  • Packages
  • Beautifier
  • Exemples / Examples
    • Des d'HTML / From HTML
    • <script type="text/javascript" src="js/toto.js"></script>
    • <script type='text/javascript' language='javascript'>
        ...
      </script>
    • background colour for "a":
      • <a onfocus="this.style.background='yellow'" onblur="this.style.background=''">
      • or, in css:
        • a:focus {background-color: yellow;}
    • redirection
      • document.location.href = http://...
    • location
      • Location object (w3schools)
      • Parsing URLs
      • location: http://example.com:8000/toto1/toto2.html#part2

        		• value
          location.href http://example.com:8000/toto1/toto2.html#part2
          location.protocol http:
          location.host example.com:8000
          location.hostname example.com
          location.port 8000
          location.pathname /toto1/toto2.html#part2 (?)
          location.hash #part2
    • relative href to a different port:
      • Relative URL to a different port number in a hyperlink?
      • http://192.168.1.10:8080/index.html 
      • // To deal with urls that imply a change of port, on the same server.
        // They begin with ":".
        document.addEventListener('click', function(event) {
          var target = event.target;
          if (target.tagName.toLowerCase() == 'a')
          {
            var parts = target.getAttribute('href').match(/^:(\d+)(.*)/);
            if (parts)
            {
              var adreca_aboluta = location.protocol+"//"+location.hostname+":"+parts[1]+parts[2];
              target.setAttribute('href', adreca_absoluta);
            }
          }
        }, false);
      • <body>
        ...
        <a href=":8000/otherpage.html">Other page on the other port at the same server</a>
        ...
        </body>
      • In this case, before changing the values, they are the following:
        • target="http://192.168.1.10:8080/:8000"
        • target.port="8080"
        • target.getAttribute('href')=":8000"
    • Simple output:
      • <p>Here is the port:
        <script type="text/javascript">
        document.write(location.port);
        </script>
        </p>
    • Alert pop-up:
      • alert("Target port is: "+target.port);
    • Debug information:
    • Emmagatzematge / Storage
    • Data / Date
    • Misc

TypeScript

DOM (Document Object Model)

Dart

Java

Gràfics vectorials / Vectorial graphics


version profile features subsets software used by samples
scripting audio video µDOM SMIL implementation authoring tools players
sync animation
SVG 1.1 full - - - -



Flame Project? (compilation)

mobile tiny (cellphone)

Laser v1? GPAC

TinyLine
basic (PDA)






1.2 full x x x x x x





mobile tiny Laser v2 GPAC

Adobe Flash
  • Adobe Flash
    • Test
    • Install
      • navegador / browser
        		 paquet del sistema / system package
        		 comprovació de la instal·lació / installation check
        		 activació / activation
        		 notes
        repo
        		 package filename
        		 package name
        		 provided file
        Chrome
        		 -
        		 -
        		 -
        		 -
        		 chrome://components/ chrome://settings/content/flash Chrome 61: Si s'activa l'opció «Pregunta-m'ho abans» (que no preguntarà mai), s'activa la llista blanca («Permet») i només aquells llocs que hi estiguin llistats tindran permís per a executar Flash.
        Firefox
        		 Adobe Flash Player flash-player-npapi-27.0.0.130-release.x86_64.rpm
        		 flash-plugin - Adobe Flash Player NPAPI
        		 /usr/lib64/mozilla/plugins/libflashplayer.so -> /usr/lib64/flash-plugin/libflashplayer.so
        		 about:addons
        • Shockwave Flash


  • Programari / Software
  • Dispositius mòbils / Mobile devices

Altres llenguatges / Other languages (parsers)

  • W3 Schools
  • SMIL (Synchronized Multimedia Integration Language) (wp)
    • SVG animation
    • A-SMIL (SMIL for Digital Signage)
    • Authoring
    • Players
  • Biblioteques / Libraries
  • Compaq's Web Language (WebL)
  • World Wide Web Wrapper Factory (W4F) (old)
  • Website Meta Language (WML) 
  • Ruby/Amazon
  • Llenguatges de marques / Markup languages
    • Comparison of document markup languages (wp)
    • Markdown versus ReStructuredText

    		•  markdown asciidoc rst
      headers # Heading 1
      [[_TOC_]]

      ## Heading 2
      ### Heading 3      		 = Document title
      :toc:
      :toclevels: 4
      :sectnums:

      == First section level 1
      === First section level 2

      == Second section level 1
      === Second section level 2       		######
      Part 1
      ######

      *********
      Chapter 1
      *********

      Section 1
      =========

      Subsection 1
      ------------

      Subsubsection 1
      ^^^^^^^^^^^^^^^

      Parapgraph 1
      """"""""""""
      table of contents [[_TOC_]] :toc:
      :toclevels: 4      		 .. toctree::
         :maxdepth: 2
         :caption: Contents:

         another_rst_file_name_without_extension
      unordered list * primer
      * segon
        * segon_primer
        * segon_segon      		 Una llista:

      * primer
      * segon
      ** segon_primer
      ** segon_segon
      ordered list 1. primer
      2. segon      		 . primer
      . segon
      negreta

      		**text en negreta**
      cursiva

      		*text en cursiva**
      format ~barrat~ [.line-through]#barrat#
      inline code format `some code` `some code`
      unicode character
      		&#128472;
      icons :white_check_mark:
      • &#x2705;
      • :white_check_mark: &#x2705;

         ... {white_check_mark} ...


      		admonition types:
      • NOTE:
      • TIP:
      • IMPORTANT:
      • CAUTION:
      • WARNING:
      blocks
      		----
      listing code
      ----

      ....
      literal
      ....

      ____
      Quote
      ____

      =====
      Example
      =====

      ****
      Sidebar
      ****      		 .. note::

         My note.
      code block ```python
      import so

      ```      		 [source, python]
      ----
      import so

      ----      		 .. code-block:: python

         import so
      language code block

      		.. py:function:: lumache.get_random_ingredients(kind=None)

         Return a list of random ingredients as strings.

         :param kind: Optional "kind" of ingredients.
         :type kind: list[str] or None
         :return: The ingredients list.
         :rtype: list[str]

      Otherwise, :py:func:`lumache.get_random_ingredients`
      will raise an exception.
      doctests
      (code snippets included in documentation that
       are executed when the documentation is built)

      		doctests
      • conf.py
        • import pathlib
          import sys
          sys.path.insert(0, pathlib.Path(__file__).parents[2].resolve().as_posix())
        • extensions = ["sphinx.ext.doctest"]
      • my_doc.srt
        • >>> lumache.get_random_ingredients()
          expected_result
      • make doctest
      • make html
      autodoc

      		Automatic documentation generation from code
      (using sphinx.ext.autodoc and docstrings in your code)
      .. autofunction:: lumache.get_random_ingredients
      .. autoexception:: lumache.InvalidKindError
      autosummary

      		(using sphinx.ext.autosummary and docstrings in your code)
      .. autosummary::
         :toctree: my_subdir_where_rst_will_be_automatically_created_by_...

         my_first_module
      block inside a list item
      		. primer
      +
      [source, bash]
      ----
      cd toto
      ----
      . segon
      +
      [source, bash]
      ----
      cd tata
      ----
      local svg image ![My diagram](/docs/my_diagram.svg) image::/docs/my_diagram.svg[My diagram]
      graphics ```mermaid

      ```      		 [mermaid]
      ....

      ....
      link to other section (default)
      		See <<_second_section>>

      == Second section
      ...
      link to other section (custom link and text)
      		See <<toto,Toto section>>
      [#toto]
      == Second section
      ...      		 See :ref:`Toto section <toto>`


      .. _toto:

      Second section
      --------------

      ...
      link to other page
      		link:my_other_page.adoc[My Other Page]
      		 :doc:`my_other_document`
      link to external url [My Site](https://mysite.org) https://mysite.org/[My Site] `My Site <https://mysite.org/>`_
      table |primera|segona|
      |-------|------|
      |1      |2     |
      |3      |4     |       		.Table title
      [%autowidth]
      |===
      |primera|segona

      |1
      |2

      |3
      |4
      |===      		 (Emacs: table.el)
      2 línies de capçaleres
      		|===
      | primera | segona

      h|tercera h|quarta

      | 1
      | 2
      |===
      colspan
      		3+|This cell spans 3 cols
      rowspan
      		.2+|This cell spans 2 rows
      list inside a cell
      		[cols="d,a"]
      |===
      |primera|segona

      |1
      |2

      * primer
      * segon

      |3
      |4

      * tercer
      * quart

      |===
      collapsible (hide)
      		[%collapsible]
      ====
      my text
      ====
      Keyboard, button, and menu macros
      		:experimental:

      From menu: menu:File[Save]
    • AsciiDoc (.adoc files)
    • Markdown (wp)
    • reStructuredText (wp)
    • Kiwi (Python)

Desenvolupament / Development

Serveis web / Web services

Message-oriented middleware
  • Info
  • Protocols
  • Implementations
    • Apache ActiveMQ
    • Apache Kafka (LinkedIn) (wp)
    • Apache Qpid (wp)
    • RabbitMQ
      • Cloud services
      • Instal·lació / Installation
        • Docker
          • rabbitmq
            • Implement RabbitMQ on Docker in 20 minutes
            • docker run -d --hostname my-rabbit --name some-rabbit -p 5672:5672 rabbitmq:3
            • docker logs some-rabbit
            • get the status, running rabbitmqctl from inside the container:
              • docker exec some-rabbit rabbitmqctl status
            • activate STOMP plugin (with web management):
              • mkdir rabbitmq-stomp
              • cd rabbitmq-stomp
              • create file Dockerfile with content:
                • FROM rabbitmq:3.8-management
                  RUN rabbitmq-plugins enable --offline rabbitmq_stomp
              • docker build -t rabbitmq-stomp .
              • docker run -d --name some-rabbit-stomp -p 15672:15672 -p 5672:5672 rabbitmq-stomp
              • http://localhost:15672/
        • Mageia
          • Mageia >5, < 5
            • urpmi rabbitmq-server
          • Mageia ==5
            • Bug 15120 - rabbitmq-server new security issues fixed upstream in 3.4.1 and 3.4.3
            • Install from download:
              • rpm --import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
              • valid versions
                • urpmi https://www.rabbitmq.com/releases/rabbitmq-server/v3.5.7/rabbitmq-server-3.5.7-1.noarch.rpm
              • invalid versions
                • urpmi https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm
                • urpmi https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.0/rabbitmq-server-3.6.0-1.noarch.rpm
                  • WARNING: rabbitmq 3.6.0-1 requires erlang "R16B03","5.10.4"; Mageia 5 provides erlang R16B02
            • dependencies
              • urpmi erlang-mnesia erlang-os_mon erlang-xmerl erlang-inets erlang-eldap erlang-public_key erlang-ssl erlang-tools erlang-compilererlang-syntax_tools erlang-crypto
          • systemctl enable rabbitmq-server.service
          • systemctl start rabbitmq-server.service
          • systemctl status rabbitmq-server.service
        • Ubuntu
          • sudo apt-get install rabbitmq-server
        • CentOS
        • AWS EC2
          • Amazon EC2 (official RabbitMQ site)
          • Problemes / Problems
            • database is stored in a dir named after the own ip address, and when the address changes (using AMI to creae new instances) the database is lost
        • Port
        • Accessible from outside (default is ipv6 ::1 only)
          • /etc/rabbitmq/rabbitmq.config (IMPORTANT: if this is the only entry in rabbit section, do not add comma at the end)
            • [
               {rabbit,
                [
                 {tcp_listeners, [{"0.0.0.0", 5672},
                                  {"::1",       5672}]}
              ...
          • from external computer:
            • nmap -p 5672 <rabbitmq_server_ip_address>
            • ...
        • Check the status
          • systemctl status rabbitmq-server
            • should return:
              • Active: active (running)
            • and not:
              • Active: active (exited)
          • rabbitmqctl status
        • Problemes / Problems
          • Troubleshooting Network Connectivity
          • resposta lenta / slow response
            • Solució / Solution
              • verify DNS resolution
                • /etc/resolv.conf
                • dig ...
          • timeout
          • check /var/log/rabbitmq/startup_{log, _err}
            • E.g.:
              • {"init terminating in do_boot",{could_not_start,rabbit,{{erlang_version_too_old,{found,"R16B02","5.10.3"},{required,"R16B03","5.10.4"}},{rabbit,start,[normal,[]]}}}}
          • rabbitmqctl ...
            • Error: unable to connect to node rabbit@localhost: nodedown
              • Solució / Solution
                • rabbitmqctl -n rabbit@ip-172-31-20-115 -p celery_vhost list_queues
          • journalctl -u rabbitmq-server / systemctl status rabbitmq-server.service
            • Error description:
                  noproc
              • Solució / Solution
                • (?) Install erlang (e.g. sudo urpmi erlang)
            • Failed to start RabbitMQ broker
              • Solució / Solution
                • setsebool -P nis_enabled 1
                • moreover, if you tried to manually start rabbitmq server (sudo /usr/lib/rabbitmq/bin/rabbitmq-server), some files and dirs may have wrong owner (root instead of rabbitmq):
                  • journalctl -u rabbitmq-server
                    • {error,{could_not_write_file,"/var/lib/rabbitmq/mnesia/rabbit@ip-xxxx/cluster_nodes.config",...
                  • sudo rm -rf /var/lib/rabbitmq/mnesia/
                  • sudo systemctl start rabbitmq-server.service
            • Event crashed log handler: (empty /var/log/rabbitmq/rabbit@...log)
              • Solució / Solution
                • update from erlang-...-R16B-03.16.el7.x86_64 -> erlang-...-R16B-03.18.el7.x86_64
      • Configuració / Setup
      • Eines / CLI Tools
        • rabbitmqctl
          • "Only root or rabbitmq should run rabbitmqctl"
          • rabbitmqctl status
          • rabbitmqctl list_queues name messages_ready messages_unacknowledged
          • rabbitmqctl list_exchanges
          • usuaris / users
            • rabbitmqctl report
            • rabbitmqctl list_users
            • rabbitmqctl add_user <username> <password>
            • rabbitmqctl add_vhost <my_vhost>
            • rabbitmqctl list_vhosts
            • rabbitmqctl set_permissions -p <my_vhost> <username> ".*" ".*" ".*"
            • set administrator privileges:
              • rabbitmqctl set_user_tags <username_administrator> administrator
          • queues
        • rabbitmqadmin
          • you need to activate rabbitmq_management plugin
          • download from github:
            • curl -O https://raw.githubusercontent.com/rabbitmq/rabbitmq-server/v3.12.x/deps/rabbitmq_management/bin/rabbitmqadmin
          • or download it from http://localhost:15672/cli/:
            • curl -O http://localhost:15672/cli/rabbitmqadmin
            • chmod +x rabbitmqadmin
          • used username has to have administration permission
          • queues
            • list queues
              • rabbitmqadmin -V celery_vhost -u <username_administrator> -p <my_password> list queues
              • rabbitmqadmin -V celery_vhost -u <username_administrator> -p <my_password> list queues <columns>
                • rabbitmqadmin -V celery_vhost -u <username_administrator> -p <my_password> list queues name messages_ready messages_unacknowledged
              • remote (e.g. AmazonMQ, where ssl is used):
                • ./rabbitmqadmin --host=b-xxx.mq.eu-west-1.amazonaws.com --ssl --vhost=my_vhost --port=15671 --username=my_user --password=my_password list queues
                  • column "messages" will contain Total number of messages (Ready + Unacked), as seen in web admin tool
                • ./rabbitmqadmin --host=b-xxx.mq.eu-west-1.amazonaws.com --ssl --vhost=my_vhost --port=15671 --username=my_user --password=my_password list queues name messages_ready messages_unacknowledged
                  • in celery:
                    • messages_ready: the number of tasks unassigned to a worker
                    • messages_unacknowledged: the number of tasks assigned to a worker, but not yet finished (see timeout, PRECONDITION_FAILED)
            • get last 10 messages
              • rabbitmqadmin -V celery_vhost -u <username_administrator> -p <my_password> get queue=<queue_name> count=10
            • purge a queue
              • rabbitmqadmin -V celery_vhost -u <username_administrator> -p <my_password> purge queue name=<queue_name>
      • Plugins
      • Used by
      • RabbitMQ Tutorials (rabbitmq-tutorials)
        • Concepts (AMQP 0-9-1)
          tutorial
          		exchange
          		 publish
          		 bind
          		 queue


          		name
          		 type
          		 routing_key
          		 routing_key
          Hello World RabbitMQ tutorial Hello World: The simplest thing that does something

          		 producer
          		 ''
          		'hello'
          		'hello'
          consumer



          		'hello'
          Work queues RabbitMQ tutorial work queues: Distributing tasks among workers (the competing consumers pattern)
          		 producer ''
          		'task_queue'
          		'task_queue'
          consumer



          		'task_queue'
          Publish / Subscribe RabbitMQ tutorial publish subscribe: Sending messages to many consumers at once
          		 producer 'logs'
          		 'fanout'


          consumer 'logs' 'fanout'
          		-
          		 'amq.gen-xxxx'
          Routing RabbitMQ tutorial routing: Receiving messages selectively
          		 producer 'direct_logs'
          		 'direct'
          		 severity

          consumer 'direct_logs' 'direct'
          		severity
          		 'amq.gen-xxxx'
          Topics RabbitMQ tutorial topics: Receiving messages based on a pattern (topics)
          		 producer 'topic_logs'
          		 'topic'
          		 x.y.z

          consumer 'topic_logs' 'topic'
          		x.y.z
          		 'amq.gen-xxxx'
          RPC  RabbitMQ tutorial rpc: Request/reply pattern example
          		 producer




          consumer




        • Python code example (using Pika)

          		 concepts
          		 code


          		producer
          		 consumer
          Hello World
          • channel
          • queue
          • exchange
          		#!/usr/bin/env python
          import pika

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          		 #!/usr/bin/env python
          import pika

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          channel.queue_declare(queue='hello') channel.queue_declare(queue='hello')
          channel.basic_publish(exchange='',
                                routing_key='hello',
                                body='Hello World!')
          print " [x] Sent 'Hello World!'"          		 print ' [*] Waiting for messages. To exit press CTRL+C'

          def callback(ch, method, properties, body):
              print " [x] Received %r" % (body,)

          channel.basic_consume(callback,
                                queue='hello',
                                no_ack=True)

          channel.start_consuming()
          connection.close()
          Work queues
          • round-robin dispatching
          • acknowledgements
          • durability
          		#!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          		 #!/usr/bin/env python
          import pika
          import time

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          channel.queue_declare(queue='task_queue', durable=True) channel.queue_declare(queue='task_queue', durable=True)
          message = ' '.join(sys.argv[1:]) or "Hello World!"
          channel.basic_publish(exchange='',
                                routing_key='task_queue',
                                body=message,
                                properties=pika.BasicProperties(
                                   delivery_mode = 2, # make message persistent
                                ))
          print " [x] Sent %r" % (message,)          		 print ' [*] Waiting for messages. To exit press CTRL+C'

          def callback(ch, method, properties, body):
              print " [x] Received %r" % (body,)
              time.sleep( body.count('.') )
              print " [x] Done"
              ch.basic_ack(delivery_tag = method.delivery_tag)

          channel.basic_qos(prefetch_count=1)
          channel.basic_consume(callback,
                                queue='task_queue')

          channel.start_consuming()
          connection.close()
          Publish / Subscribe
          • exchange
            • direct
            • topic
            • header
            • fanout
          • temporary queues
          • binding
          		#!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          		 #!/usr/bin/env python
          import pika

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          channel.exchange_declare(exchange='logs',
                                   type='fanout')          		 channel.exchange_declare(exchange='logs',
                                   type='fanout')

          		result = channel.queue_declare(exclusive=True)
          queue_name = result.method.queue

          		channel.queue_bind(exchange='logs',
                             queue=queue_name)
          message = ' '.join(sys.argv[1:]) or "info: Hello World!"
          channel.basic_publish(exchange='logs',
                                routing_key='',
                                body=message)
          print " [x] Sent %r" % (message,)          		 print ' [*] Waiting for logs. To exit press CTRL+C'

          def callback(ch, method, properties, body):
              print " [x] %r" % (body,)

          channel.basic_consume(callback,
                                queue=queue_name,
                                no_ack=True)

          channel.start_consuming()
          connection.close()
          Routing
          • exchange
            • direct
          		#!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          		 #!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          channel.exchange_declare(exchange='direct_logs',
                                   type='direct')          		 channel.exchange_declare(exchange='direct_logs',
                                   type='direct')

          		result = channel.queue_declare(exclusive=True)
          queue_name = result.method.queue

          		severities = sys.argv[1:]
          if not severities:
              print >> sys.stderr, "Usage: %s [info] [warning] [error]" % \
                                   (sys.argv[0],)
              sys.exit(1)

          for severity in severities:
              channel.queue_bind(exchange='direct_logs',
                                 queue=queue_name,
                                 routing_key=severity)
          severity = sys.argv[1] if len(sys.argv) > 1 else 'info'
          message = ' '.join(sys.argv[2:]) or 'Hello World!'
          channel.basic_publish(exchange='direct_logs',
                                routing_key=severity,
                                body=message)
          print " [x] Sent %r:%r" % (severity, message)          		 print ' [*] Waiting for logs. To exit press CTRL+C'

          def callback(ch, method, properties, body):
              print " [x] %r:%r" % (method.routing_key, body,)

          channel.basic_consume(callback,
                                queue=queue_name,
                                no_ack=True)

          channel.start_consuming()
          connection.close()
          Topics
          • exchange
            • topic
          		#!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          		 #!/usr/bin/env python
          import pika
          import sys

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))
          channel = connection.channel()
          channel.exchange_declare(exchange='topic_logs',
                                   type='topic')          		 channel.exchange_declare(exchange='topic_logs',
                                   type='topic')

          		result = channel.queue_declare(exclusive=True)
          queue_name = result.method.queue

          		binding_keys = sys.argv[1:]
          if not binding_keys:
              print >> sys.stderr, "Usage: %s [binding_key]..." % (sys.argv[0],)
              sys.exit(1)

          for binding_key in binding_keys:
              channel.queue_bind(exchange='topic_logs',
                                 queue=queue_name,
                                 routing_key=binding_key)
          routing_key = sys.argv[1] if len(sys.argv) > 1 else 'anonymous.info'
          message = ' '.join(sys.argv[2:]) or 'Hello World!'
          channel.basic_publish(exchange='topic_logs',
                                routing_key=routing_key,
                                body=message)
          print " [x] Sent %r:%r" % (routing_key, message)          		 print ' [*] Waiting for logs. To exit press CTRL+C'

          def callback(ch, method, properties, body):
              print " [x] %r:%r" % (method.routing_key, body,)

          channel.basic_consume(callback,
                                queue=queue_name,
                                no_ack=True)

          channel.start_consuming()
          connection.close()
          RPC
          • callback queue
          • correlation_id
          • reply_to
          		#!/usr/bin/env python
          import pika
          import uuid

          class FibonacciRpcClient(object):
              def __init__(self):
                  self.connection = pika.BlockingConnection(pika.ConnectionParameters(
                          host='localhost'))

                  self.channel = self.connection.channel()

                  result = self.channel.queue_declare(exclusive=True)
                  self.callback_queue = result.method.queue

                  self.channel.basic_consume(self.on_response, no_ack=True,
                                             queue=self.callback_queue)

              def on_response(self, ch, method, props, body):
                  if self.corr_id == props.correlation_id:
                      self.response = body

              def call(self, n):
                  self.response = None
                  self.corr_id = str(uuid.uuid4())
                  self.channel.basic_publish(exchange='',
                                             routing_key='rpc_queue',
                                             properties=pika.BasicProperties(
                                                   reply_to = self.callback_queue,
                                                   correlation_id = self.corr_id,
                                                   ),
                                             body=str(n))
                  while self.response is None:
                      self.connection.process_data_events()
                  return int(self.response)

          fibonacci_rpc = FibonacciRpcClient()

          print " [x] Requesting fib(30)"
          response = fibonacci_rpc.call(30)
          print " [.] Got %r" % (response,)

          		 #!/usr/bin/env python
          import pika

          connection = pika.BlockingConnection(pika.ConnectionParameters(
                  host='localhost'))

          channel = connection.channel()

          channel.queue_declare(queue='rpc_queue')

          def fib(n):
              if n == 0:
                  return 0
              elif n == 1:
                  return 1
              else:
                  return fib(n-1) + fib(n-2)

          def on_request(ch, method, props, body):
              n = int(body)

              print " [.] fib(%s)"  % (n,)
              response = fib(n)

              ch.basic_publish(exchange='',
                               routing_key=props.reply_to,
                               properties=pika.BasicProperties(correlation_id = \
                                                               props.correlation_id),
                               body=str(response))
              ch.basic_ack(delivery_tag = method.delivery_tag)

          channel.basic_qos(prefetch_count=1)
          channel.basic_consume(on_request, queue='rpc_queue')

          print " [x] Awaiting RPC requests"
          channel.start_consuming()
    • Mosquitto
      • Info
      • Instal·lació / Installation
      • Ús / Usage
        • servidor / server
          • systemctl enable mosquitto.service
          • systemctl start mosquitto.service
        • client
          • producer
            • mosquitto_pub -h <broker> -t <topic> -m <message>
            • rtl_433 -F "mqtt://localhost:1883"
          • consumer
            • mosquitto_sub -h <broker> -t <topic>
            • subscribe to all topics
              • mosquitto_sub -v -t '#'
      • ...
    • ...

http://www.francescpinyol.cat/www.html
Darrera modificació: 16 de desembre de 2024 / Last update: 16th December 2024

Valid HTML 4.01!

Cap a casa / Back home.