Fabian M. Suchanek
Course material adapted from Antoine Amarilli
http://a3nm.net
The Internet
and Pierre Senellart
http://pierre.senellart.com
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
2
1969
ARPANET (ancêtre d'Internet)
1974
Transmission Control Protocol (TCP)
(Vinton G. Cerf & Robert E. Kahn, Turing award 2004)
1990
World Wide Web, HTTP, HTML (Tim Berners-Lee, Robert Cailliau)
1993
Mosaic (premier navigateur graphique à succès,
ancêtre de Netscape)
1994
Yahoo! (David Filo, Jerry Yang)
1994
Fondation du W3C
1995
Amazon.com, Ebay
1995
Internet Explorer
1995
AltaVista (Louis Monier, Michael Burrows)
1998
Google (Larry Page, Sergey Brin)
2001
Wikipedia (Jimmy Wales)
2004
Mozilla Firefox
2005
YouTube
Un bref historique du Web
3
L'Internet, visualisé
4
• Près de 150 millions de
domaines
, dont 75% dans
.com
• En 2002, >50% du contenu est en
anglais
, 6% en français.
• Plus de 2 milliards d'
utilisateurs
sur Internet.
• Google connaît plus d'un
trillion
(1E12) d'URLs uniques.
=>
On soupçonne qu'une large partie du Web n'est pas indexable :
le
Web caché
.
(source)
(source)
(source)
(source)
Statistiques
5
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
6
•
Le client
(navigateur: Internet Explorer, Firefox, Safari)
• demande au serveur des informations
• affiche des pages pour l'utilisateur
•
Le serveur
(Apache, Microsoft IIS)
• reçoit en permanence les requêtes des clients
• renvoie les documents correspondants
Architecture client-serveur
Client
Serveur
7
important
Un
navigateur
est un logiciel qui permet
d'obtenir et afficher des pages Web.
• Les plus connus sont: Internet Explorer, Chrome, Safari
• Dépend du
système d'exploitation sous-jacent
• Fonctionne sur un
ordinateur
(ou téléphone).
Le navigateur prend le rôle du
client
.
Navigateur
Client
8
Mosaic. Premier navigateur graphique répandu,
1993-1997
.
• De
80%
en 1994 à
< 10%
en 1996.
Netscape. Lancé en
1994
, basé sur Mosaic.
• Propriétaire, usage non-commercial
gratuit
.
•
80%
d'usage en 1996,
<10%
en 2001.
• N'est
plus maintenu
depuis 2008.
Internet Explorer. Lancé en
1995
, basé sur
Spyglass Mosaic
.
• Propriétaire, fourni avec
Windows 95
.
• IE 6 lancé en 2001 atteint
80%
du marché.
• Mène au procès
antitrust
aux USA, 1998-2001.
Firefox. Lancé en
2002
•
Libre
et gratuit.
• Navigation à
onglets
.
• Attaque le
monopole
de IE 6.
(details)
Navigateurs Web historiques
9
Évolution
Source: StatCounter.
10
Navigateurs Web historiques
11
IE
. Version actuelle est
IE 11
Firefox
. Toujours
dans la course
.
Safari
. Lancé en 2003, navigateur par défaut de
Mac OS X
(succédant à Netscape puis IE sous Mac OS).
Opera
. Lancé en
1996
, mais
payant
jusqu'en 2000 et financé par la
publicité
jusqu'en 2005.
Maintenant
gratuit
mais propriétaire.
Chrome
. Lancé en
2008
par
Google
.
Propriétaire mais version libre également disponible.
Mobile
. Important ! Safari sur
iOS
, Android browser ou
Chrome sur
Android
. Firefox mobile,
et autres (Blackberry, Opera Mini, Opera Mobile...)
Navigateurs Web actuels
12
Répartition récente
Source: StatCounter (septembre 2013).
13
CC by-sa
Roke, Altes, and Peeperman
Répartition géographique
14
Un
moteur de rendu
(rendering engine) est un logiciel
qui affiche une page Web.
Trident
. Internet Explorer seulement, propriétaire.
Gecko
. Firefox et apparentés, libre et gratuit.
KHTML
. Libre et gratuit, projet KDE.
WebKit
. Fork de KHTML, Safari et (anciennement) Chrome,
navigateurs Android et iOS, libre et gratuit.
Blink
. Fork récent (avril 2013) de Webkit, Chrome, Opera.
Presto
. Propriétaire, anciennes versions d'Opéra.
Autres
. Dillo, autres moteurs exotiques...
Moteurs de rendu
15
Navigateurs textuels
Aussi :
navigateurs auditifs
pour les aveugles...
Try it out!
Simule ce qu'un bot voit
16
•
Crawlers
pour moteurs de recherche (indexation) :
Googlebot (Google), bingbot (Bing)...
•
Crawlers
plus obscurs : meanpathbot,
ahrefsbot, ezoomsbot, Career-X bot, msnbot,
news bot, seznambot, ChangeDetection bot ? ...
•
Lecteurs RSS
et agrégateurs.
•
Harvesters
d'adresses email.
• Utilisateurs d'
API
.
Robots
Nombreux
programmes automatisés
sur le Web :
17
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
18
A URL (uniform resource locator) is a string that identifies
a document on the internet.
http://
en.wikipedia.org
/wiki/Telecom_ParisTech
#
History
This (usually) refers to a part of a document
that is physically stored on a computer (the server).
URL
protocol
computer
path
fragment
19
Where is this computer?
(or: host)
important
An
Internet Protocol address
(IP address) is a numerical label
assigned to each device (e.g., computer, printer) participating
in a computer network that uses the Internet Protocol.
(The IP address is like the phone number of the host computer)
The host part of a URL corresponds to an IP address.
[Wikipedia/IP address]
http://
en.wikipedia.org
/wiki/Telecom_ParisTech
#
History
There are two versions of IP addresses:
•
IPv4
(old version)
•
IPv6
(new version)
IP Address
20
2001:db8:0:1234:0:567:8:1
important
Original version of the addresses:
IP version 4
(IPv4)
• consists of 4 bytes (32 bits), e.g.
127.255.255.255
• allows
= 4 billion addresses
• All addresses were taken in February 2011!
New version :
IP version 6
(IPv6)
• consists of 16 bytes (128 bits), e.g
2001:0db8:85a3:0042:1000:8a2e:0370:7334
• allows
addresses
• should be sufficient for current human population
IP Address: IPv4 and IPv6
21
A
name server
is a computer that maintains a mapping
from host names to IP addresses.
There are 13 such name servers, plus their clones:
Finding the computer for the address
22
telecom-parsitech.fr = 123.456.789.0123
en.wikipedia.org = 0123.123.456.789
...
• Name servers replicate their data on other name servers
=>
The name servers are like a global distributed database
• The internet service providers usually also maintain a copy
• If the host name is unknown, requests are forwarded
to another name server
• The name servers have a fixed, known IP address
• The entire structure is called the
Domain Name System
(DNS)
• Registration of a hostname for an IP address is
done by
domain name registrars
• These have to be accredited by the Internet Corporation
for Assigned Names and Numbers (
ICANN
)
Finding the computer
23
important
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
24
Client
Server
25
The Internet Protocol Suite
abstract, virtual
communication
real, physical
communication
intermediate
abstraction
layers
Client
Server
26
The Internet Protocol Suite
abstract, virtual
communication
intermediate
abstraction
layers
(see Wikipedia)
Client
Server
27
The Internet Protocol Suite
Application Layer
Transport Layer
Internet Layer
Link Layer
Protocols:
HTTP,FTP,
POP, SSH,
...
Protocols:
TCP, UDP; ...
Protocols:
IPv4, IPv6, ...
Protocols:
DSL, ISBN,
Ethernet, ...
28
Server
Client
The Internet Protocol Suite
important
A protocol is a standardized set of rules that describe
how information is transmitted, for example in a network
such as Internet between a client and a server.
How are
you doing?
Awesome!
How are
you doing?
Awesome!
Protocol
29
Hi!
Hi!
(content here)
HTTP
: HyperText Transfer Protocol, most widely used
protocol on the World Wide Web. Allows a client to say
which Web page it wants, and a server to respond with that page.
HTTPS
: Like HTTP, but with encryption and authentification.
FTP
: For transmission of files. Used sometimes on the Web
for the transmission of large files.
SSH
: Cryptographic protocol for secure data communication,
remote command-line login, remote command execution, etc.
Protocols:
HTTP,FTP,
POP, SSH,
...
Application Layer
30
Application Layer Protocols
Server
Client
important
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
31
HTTP is an
application protocol
at the basis of the World
Wide Web. The latest and most widely used version is
HTTP/1.1
.
Client
request
:
GET /MarkUp/ HTTP/1.1
Host: www.w3.org
Server
response
:
HTTP/1.1 200 OK
...
Content-Type: text/html;
<html> ...
• Two main HTTP methods: GET and POST (HEAD is also used in
place of GET, to retrieve meta-information only).
• Additional headers, in the request and the response
• Possible to send parameters in the request (key/value pairs).
HTTP Protocol
32
important
• Simplest type of request.
• Possible parameter are sent at the end of a URL, after a “?”
• Not applicable when there are too many parameters,
or when their values are too long.
• Method used when a URL is directly accessed in a browser,
when a link is followed, and for some forms.
Try it out!
wget -S -O <file> <url>
URL:
http://www.google.com/search?q=hello
Corresponding HTTP GET request:
GET /search?q=hello HTTP/1.1
Host: www.google.com
HTTP, Client, GET
Example:
33
• Method only used for submitting (longer) forms.
POST /php/test.php HTTP/1.1
Host: www.w3.org
Content-Type: application/x-www-form-urlencoded
Content-Length: 100
type=search&title=The+Dictator&format=long&country=US
Example:
34
HTTP, Client, POST
• Web clients can identify themselves in the request
with a character string
• Useful to serve
different content
to
different browsers, detect robots
• but any client can say it's any other client!
• Historical confusion on naming: all common browsers
identify themselves as Mozilla!
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fr;
rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy)
Firefox/3.0.3
35
HTTP, Client, Identification
• A Web client can specify in the request to the Web server:
• the
content type
it can process
(text, images, multimedia content), with preferrence indicators
• the
languages
preferred by the user
• The Web server can thus propose different file formats,
in different languages.
• In practice, content negociation on the language works,
and is used, but content negociation on file types does not
work because of bad default configuration of some browsers.
HTTP, Client, Content negotiation
Accept: text/html,application/xhtml+xml,application/xml;
q=0.9,*/*;q=0.8
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
36
• A client can ask for downloading a page only if it has been modified
since some given date.
• Most often not applicable, the server giving rarely a reliable
last modification date (difficult to obtain for dynamically
generated content!).
If-Modified-Since: Wed, 15 Oct 2008 19:40:06 GMT
304 Not Modified
Last-Modified: Wed, 15 Oct 2008 19:20:00 GMT
37
HTTP, Client, Conditional downloading
• When a Web browser follows a link or submits a form,
the client transmits the originating URL
to the destination Web server.
• Even if it is not on the same server!
Example: Visitors of the dancing class Web page
came from:
38
HTTP, Client, Referer
• The client can ask for only a portion of the file
• This is useful if the download is interrupted
• This feature was misused in 2011 to bring down
Apache servers
Range: bytes=0-42
39
HTTP, Client, Range
• HTTP allows for protecting access to a Web site
by an
identifier
and a
password
• Attention: (most of the time) the password goes
through the network uncrypted (but for instance, just
encoded in Base64, revertible encoding)
•
HTTPS
(variant of HTTP that includes encryption,
cryptographic authentication, session tracking, etc.) can be
used instead to transmit sensitive data
HTTP, Client, Authentication
GET ... HTTP/1.1
Authorization: Basic dG90bzp0aXRp
40
important
• By default, parameters are sent (with GET or POST) in the form:
name1=value1&name2=value2
and special characters (accented characters, spaces) are replaced
by codes such as
+
and
%20
. This way of sending parameters is
called
application/x-www-form-urlencoded
.
• For the POST method, another heavier encoding can be used
(several lines per parameter), similar to the way emails are built:
mostly useful for sending large quantity of information. Encoding
named
multipart/form-data
.
Parameter encoding
41
HTTP/1.1 200 OK
...
Content-Type: text/html;
<html> ...
HTTP, Server Response
Status code
Header
Body (= content)
42
1xx Information
2xx Success
200 : OK
3xx Redirection
301 : permanent
302 : temporary
4xx Client error
400 : syntax error
401 : authentification needed
403 : forbidden
404 : not found
5xx Server error
500 : Internal server error
HTTP, Server, Status
43
• The browser behaves differently depending on the
content type
returned: display a Web page with the layout engine
display an image, load an external application, etc.
•
MIME
classification of content types (e.g.,
image/jpeg
,
text/plain
,
text/html
,
application/xhtml+xml
,
application/pdf
etc.)
• For a HTML page, or for text, the browser must also know what
character set
is used (this has precedence over
the information contained in the document itself)
• Also returned: the content length
(can be used to display a progress bar)
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 3046
HTTP, Server, Content type
44
A
cookie
(in networking jargon) is Information in the form
of key/value pairs that a Web server asks a Web client to keep and
retransmit with each HTTP request (for a given domain name).
• Can be used to keep information on a user as she is visiting a
Web site, between visits, etc.: electronic cart, identifier, and so on.
• Practically speaking, most often only stores a
session identifier
,
connected, on the server side, to all session information
(connected or not, user name, data)
• Simulates the notion of session, absent from HTTP itself
HTTP, Server, Cookies
Set-Cookie: session-token=RJYBsG//azkfZrRazQ3SPQhlo1FpkQka2;
path=/; domain=.amazon.de;
expires=Fri Oct 17 09:35:04 2008 GMT
Cookie: session-token=RJYBsG//azkfZrRazQ3SPQhlo1FpkQka2
45
important
A
Proxy
is an intermediate server
between client and server.
• Can be used on
client
side or
server
side.
• Applications:
•
Filter
or
censor
the Web
(employers; schools, totalitarian regimes, parental control, etc.)
• Keep a
diary
of activity
• Keep a
cache
(across several users)
•
Anonymisation
to hide the true origin of the request
• Other
modifications
: translation, etc.
Proxies
46
Introduction
Browsers
IP Addresses
Internet Protocol Suite
HTTP
Other Protocols
Overview
47
Application Layer
Transport Layer
Internet Layer
Link Layer
Protocols:
HTTP,FTP,
POP, SSH,
...
Protocols:
TCP, UDP; ...
Protocols:
IPv4, IPv6, ...
Protocols:
DSL, ISBN,
Ethernet, ...
48
Server
Client
The Internet Protocol Suite
Transport Layer
TCP Protocol
• Establish a connection between client and server
• Transmit a sequence of bytes,
split into
packets
• Make sure the packets arrive in the
right order
• Get
confirmation
from the other side
•
Retransmit
lost packages
• Check for
errors
in the packages by a checksum
• Control for
congestion
The
Transmission Control Protocol
(TCP) provides reliable, ordered,
error-checked delivery of a stream of bytes between client and server.
49
TCP
Application Layer
Transport Layer
Internet Layer
Link Layer
Protocols:
HTTP,FTP,
POP, SSH,
...
Protocols:
TCP, UDP; ...
Protocols:
IPv4, IPv6, ...
Protocols:
DSL, ISBN,
Ethernet, ...
50
Server
Client
The Internet Protocol Suite
Internet Layer
Protocols:
IPv4, IPv6, ...
The
Internet Protocol
(IP) has the task of delivering packets
from the source host to the destination host solely based
on the IP addresses in the packet headers.
51
Internet Protocol (IP)
Client
Server
Intermediate routers
Application Layer: HTTP
Transport Layer: TCP
Internet Layer: IP
52
Server
Client
Summary: Internet Protocol Suite