Ajaxtest Architecture and Operational Overview
Eric L. Edberg - 6/2014
Ajaxtest Feature
Description
Ajaxtest is a JavaScript tool that loads and executes in a web browser on laptops and mobiles. It downloads a sequence of pre-defined files using JavaScript’s “xmlhttprequest” method from 1 or more content servers located in the lab or internet. After each file download it logs test results into a server-based log file and MySQL database. At the completion of a test, transfer analytics are displayed that include a summary of each test performed, an analysis of throughput change between optimized and non-optimized flows, and a graphical histogram showing ranges of throughput normalized for each file size and server pair.
Ajaxtest supports the following capabilities:
· Download (GET) and upload (PUT) binary data transfer using JavaScript’s xmlhttprequest method
· Transfer using either HTTP or HTTPS protocols
· Supports downloading of pre-defined .html ASCII or binary files, images, or videos
· Optional data exclusion of test outliers using Std Dev, Mean or Median methods to drop invalid results
· Logging of test results into a back-end server in text log files and back-end MySQL database
· Documenting additional test setup and condition parameters
· Ability to define an abort timer to catch stalled or hung transfers
· Data isolation to segregate and secure vendor tests
· Test Summary, Gain Summary, Transfer Metrics & Histogram, and Throughput Analytics are auto-calculated and displayed in the browser at the conclusion of each test
· Ajaxtest Query & Analytics program to query for complex criteria, extract results from the MySQL database and re-calculate results (similar to those displayed at the conclusion of each test)
CMDS Provisioning
Assumptions
The CMDS system must be provisioned to “exclude” TCP optimization to specific TCP addresses hosted by the Core Content Servers. CMDS automatically optimizes HTTP traffic over port 80 except when rules are added to exclude specific IP addresses. Each Core Content Server has 1 IP address dedicated to No TCP Optimization flow that must be added to the CMDS ACL and ACR lists to exclude optimization.
· San Jose
o 206.217.192.73
· Wash, DC
o 67.213.216.182
· Sydney, AU
o 101.0.98.221
· Lab IIS
o 10.168.207.107
· Lab Linux
o 10.168.207.109
Core Content
Servers
Ajaxtest requires and downloads content from well-known content servers. These servers are leased by the CMDS organization from service providers and provisioned / managed by the CMDS team (Eric Edberg).
There are specific requirements a server must meet:
· The server must support a minimum of 2 IP addresses on the same physical Ethernet interface. This allows 1 IP address to be provisioned by CMDS to support TCP Optimization and the second IP support the No TCP Optimization flow. This ensures that the exact same logical path to the content is used.
· The server provider must not meter or cap bandwidth at a maximum or sustained average rate.
· Specific TCP options must be enabled such as:
o TCP Timestamp enabled
o TCP Offloading turned off
· Must allow “root” privileges on the server to support executing tcpdump, interface provisioning and other diagnostic commands
· Support the installation and customization of numerous packages such as:
o apache
o MySQL
o PHP
In addition to the 3 Core Content Servers, 2 additional severs are leased that host the core Ajaxtest application, MySQL database and log files. The functionality of these servers could also be hosted on the core content servers if needed. These servers are provided by Hostmonster and are considered “shared hosting” and therefore not suitable for controlled (timed) transfer testing due to the variability of shared-hosted environment. They do not restrict total monthly download rate limits and are often used to stream very large file transfers during long-term reliability tests.
o Internet primary Ajaxtest server
o Internet backup Ajaxtest server
o Integration / Test Server
· http://cmds.web.alcatel-lucent.com/Ajaxtest
· http://10.168.207.106/Ajaxtest
o Lab Ajaxtest Server
Internet Content
Servers
The following internet-based content servers are supported. Each of the servers are a stand-alone Quad-Core computer with the following capabilities:
· CentIOS operating system
· 1TB disk
· >= 4G memory
· dedicated 1G Ethernet interface
· At least 2 dedicated IP addresses
· Other requirements and capabilities exist
Internet Content Server Locations
· San Jose, CA
o West Coast
o 100TB.com
· Wash, DC
o East Coast
o 100TB.com
· Sydney, AU
· CMDS1.com
o Hostmonster
· CMDS2.com
o Hostmonster
Lab Content Servers
· CMDS-DMZ
o IIS
o Dell
· CMDS-18
o Ubuntu Linux
o Dell
Ajaxtest MySQL
Database & File Logging
After each file transfer the program immediately logs the results of the transfer into a logfile and MySQL database located in the internet or lab.
The logfile is stored in a common logfile for all historical tests (large) and a unique logfile for each test that was performed. This is an example of a logfile on 6/2014. New fields are always appended as a new column when required to preserver columnar index.
Date ClientIP
DateEpoch TestNum Hostname
UE_Type FileSize TimeMS TimerMS
URL
Protocol Comment FileName
UEModel Location TCPOpt TestID DelayWAN DelayRAN
JitterRAN
CellProximity AttenuationRF
RTTMeasuredMS Tester
2013-05-27 21:34:34
75.205.40.89 1369690472286
1
75.205.40.89 FOA-3G
512000 1376
1376 http://67.20.117.180/ftp/Content/500K.html
XMLHTTP Test from home.
500K.html USB-760 Lenexa,
KS
0
130527213300
flattum
2013-05-27 21:34:35
75.205.40.89 1369690474365
2
75.205.40.89 FOA-3G
512000
563 563 http://67.20.117.180/ftp/Content/500K.html
XMLHTTP Test from home.
500K.html USB-760 Lenexa,
KS
0
130527213300
flattum
Ajaxtest Form
Option Summary
The Ajaxtest program provide numerous form options that control program operation.
· Detailed explanation about each form field are shown when you mouse-over a form field
Required Form
Parameters
Optional Form
Parameters
The following are optional form parameters.
Ajaxtest Query
& Analytics
The Ajaxtest Query / Analytics program allows users to query for previous test results and output results into numerous formats:
· XLS Export
· Test Summary Report
· Test Summary Results Report
· Analytics Report
· Throughput Scatter plot
The query also allows the user to design a query to exclude test outliers that may be due to invalid result outliers e.g.: failed or stalled transfers. After initially reviewing raw results the user may provisioned the form to automatically exclude outliers based on STD DEV, Mean or Median algorithms.
Vendor Security
Isolation
Ajaxtest isolates test results dependent on the user who is logged into the system. Each vendor is supplied a dedicated login that they use when testing. The test form and analytics programs are coded to only allow the vendor to see, export and/or download results associated to their company.
Vendors logins include:
· cmds (Alcatel-Lucent )
o authorized to view all test results
· verizon
· telstra
· contextream
· The Vendor Partner in the form above is automatically managed in these cases
Ajaxtest Analytics
Gain Calculation
The Time and Throughput Gain is calculated using the formulas below:
· xAvgNOOPT == average time or throughput it took relative to the No TCP Optimization flow
· xAvgOPT == averate time or throughput it took relative to the TCP Optimization flow
When analyzing TIME
· var xGain = ((xAvgNOOPT-xAvgOPT)/xAvgNOOPT)*100;
When analyzing THROGHPUT
· var xGain = ((xAvgOPT / xAvgNOOPT)-1)*100;
Summary Statistics
Gain Statistics
Metrics Summary
Statistics
Metric summary statistics are calculated using the TIME it took to transfer the file. The bar widths and buckets are normalized to accurately visually compare results.
· In this example you can easily identify the wider spread of the blue No TCP Optimized flow compared to the red TCP Optimized flow
· CMDS clearly optimized the TCP flow buffers and decreased the time it took to download the test file from the test server
· This is a classic example of CMDS TCP Optimization behavior
· In this test there were no test outliers which would have presented themselves as data points taking a long time. If this was detected then re-executing Ajaxtest Query / Analytics using one of the exclusion algorithms may have been appropriate
Throughput XY Plot
Ajaxtest displays an X-Y plot at the conclusion of each test that’s shows the transfer throughput in b/s for each server. The plots show the throughput for all files transferred to/from the server e.g.: multiple file transfers are included in the same plot. Using you mouse, you can select ranges and zoom into specific data points (similar file sizes). Multiple X-Y plots are shown, one for each test server.
Ajaxtest Query –
Test Summary Report
Ajaxtest Query includes a new Test Summary Results report that generates a summary for each test. The results are suitable to import into an XLS for gain comparison analysis.
Test Summary Report
Example
Ajaxtest Excluding
Outliers
CMDS Qwiki -> CMDS Tools -> Ajaxtest -> Analytics - Excluding Outliers