@EyalG: You could probably infer a 304 vs 200 hit if the transferSize is minimal but the decodedSize is large (and responseStart/responseEnd would probably be equal or minimal).

Not sure if there are any other codes you could determine or guess, unless you have ideas?

is there anyway to determine the response code by transfer size / decodedSize ?
thanks!

many thx!

In the meantime I have found the decompression javascript on github, and found it very readable, It is all there, e.g. “*1” being just a separator between timings and sizes, just as you say.

I have already successfully tested that in R and it works great.

Thx again, this is really great stuff,
Wolfgang

Hi Wolfgang!

Those are likely transfer sizes — see the resourcetiming-compression.js documentation here:
https://github.com/nicjansma/resourcetiming-compression.js#resource-sizes

many thx for sharing the details to the compression, which I found very helpful to unravel raw data from mpulse except for one small detail: sometimes there is a star in the base36 strings like “8*1mh” which I do not find in the article and I guess there is a special meaning to this.

Would be great if you could provide some hint about how this is supposed to be handled.

Best regards,
Wolfgang

Hi Scott,

Currently here: https://github.com/lognormal/boomerang/pull/44

And waiting on me to integrate them into the new Mocha tests.

Has ResourceTiming Compression been added to the latest version of Boomerang yet? I don’t seem to find the code in the GitHub repository. What source file is this included in?

Scott

Thanks for the ideas, those are both great improvements.

For the deltas version, the only concern I have is using it when re-constructing the complete Waterfall. For most cases of RT we’ve seen in the wild, the end/start timestamps line up (eg domainLookupEnd==connectStart, connectEnd==requestStart), but if they’re slightly off (say the browser didn’t immediately send the request after TCP connection, so connectEnd is 10 but requestStart is 50), then you’d be missing the fact that the overall duration is 40ms longer than the deltas suggest.

I’ll do a bit of in-the-wild analysis to see if this is prevalent or not. I’d also be worried about future implementations (eg FF and maybe even Safari some day) having this issue and it not being obvious.

The Key vs Delimiter approach is great. With the RT data we’ve seen so far, we’ve found it very rare for 0-in-the-middle timestamps, but you do suggest one (probably somewhat common) example if the resource was redirected to different content on the same domain. I like using the keys instead of a fixed array, and whether you’re doing “deltas” or all of the timestamps, you could use the same approach.

I found two additional techniques for compressing the data further.

1. Deltas – Rather than send back every time property’s value, I convert them to deltas. For example, you have this structure with 12 array elements:

“name”: [initiatorType,
startTime,
responseEnd,
responseStart,
requestStart,
connectEnd,
secureConnectionStart,
connectStart,
domainLookupEnd,
domainLookupStart,
redirectEnd,
redirectStart]

Here’s the “deltas” version of the same info:

“name”: [initiatorType,
startTime,
content, // responseEnd – responseStart
ttfb, // responseStart – requestStart
connect, // connectEnd – connectStart
ssl, // secureConnectionStart – connectStart
dns, // domainLookupEnd – domainLookupStart,
redir] // redirectEnd – redirectStart

That reduces it from 12 array elements to 8.

2. Key vs delimiter – As you mention, many of the values are zero. The problem with an array is that if the *latter* values are non-zero, all the intermediate zeroes *must* be specified to preserve position. Since we later convert the array to a string, rather than using a delimiter character (such as comma) I use single character keys – “C” for “content”, “T” for “ttfb”, etc. That way, it’s not necessary to preserve order, and all zero values can be left out.

For example, in your approach if there’s a 12ms redirect but no time is spent on DNS or connect, the array is [“script”, 252, 71, 48, 0, 0, 0, 0, 0, 0, 0, 12, 0] which converts to the string “370,1z,1c,0,0,0,0,0,0,0,c”. The “deltas” approach array is [“script”, 252, 23, 48, 0, 0, 0, 12] which converts to the string “370CnT1cRc” – 10 chars vs 25.

Note 1: I user upper case chars for my delimiter keys so they don’t get confused with the base 36 alpha characters.

Note 2: In my approach I add a 9th element to the array: blocking. This is needed if you want to construct a waterfall.

I bet there are a few places where these additional optimizations mess up. Given SOASTA’s huge amount of experience with RUM, it’d be great to hear your feedback on whether these are worthwhile.