When advocating a new technology I always try to use it in the way that real world developers will, and for WebVR (the VR-only precursor to WebXR), building a game is currently one of the best ways to do that. So for the winter holidays I built a game, Jingle Smash, a classic block tumbling game. If you haven't played it yet, put on your headset and give it a try. Now an overview of how I built it.
This article is part of my ongoing series of medium difficulty ThreeJS tutorials. I’ve long wanted something in between the intro “How to draw a cube” and “Let’s fill the screen with shader madness” levels. So here it is.
ThreeJS
Jingle Smash is written in ThreeJS using WebVR and some common boilerplate that I use in all of my demos. I chose to use ThreeJS directly instead of A-Frame because I knew I would be adding custom textures, custom geometry, and a custom control scheme. While it is possible to do this with A-Frame, I'd be writing so much code at the ThreeJS level that it was easier to cut out the middle man.
Physics
Jingle Smash is an Angry Birds style game where you lob an object at blocks to knock them over and destroy targets. Once you have destroyed the required targets you get to the next level. Seems simple enough. And for an 2D side view game like Angry Birds it is. I remember enough of my particle physics from school to write a simple 2D physics simulator, but 3D collisions are way beyond me. I needed a physics engine.
After evaluating the options I settled on Cannon.js because it's 100% Javascript and has no requirements on the UI. It simply calculates the positions of objects in space and puts your code in charge of stepping through time. This made it very easy to integrate with ThreeJS. It even has an example.
Graphics
In previous games I have used 3D models created by an artist. For this Jingle Smash I created everything in code. The background, blocks, and ornaments all use either standard or generated geometry. All of the textures except for the sky background are also generated on the fly using 2D HTML Canvas, then converted into textures.
I went with a purely generated approach because it let me easily mess with UV values to create different effects and use exactly the colors I wanted. In a future blog I'll dive deep into how they work. Here is a quick example of generating an ornament texture:
const canvas = document.createElement('canvas')
canvas.width = 64
canvas.height = 16
const c = canvas.getContext('2d')
c.fillStyle = 'black'
c.fillRect(0, 0, canvas.width, canvas.height)
c.fillStyle = 'red'
c.fillRect(0, 0, 30, canvas.height)
c.fillStyle = 'white'
c.fillRect(30, 0, 4, canvas.height)
c.fillStyle = 'green'
c.fillRect(34, 0, 30, canvas.height)
this.textures.ornament1 = new THREE.CanvasTexture(canvas)
this.textures.ornament1.wrapS = THREE.RepeatWrapping
this.textures.ornament1.repeat.set(8, 1)
Level Editor
Most block games are 2D. The player has a view of the entire game board. Once you enter 3D, however, the blocks obscure the ones behind them. This means level design is completely different. The only way to see what a level looks like is to actually jump into VR and see it. That meant I really needed a way to edit the level from within VR, just as the player would see it.
To make this work I built a simple (and ugly) level editor inside of VR. This required building a small 2D UI toolkit for the editor controls. Thanks to using HTML canvas this turned out not to be too difficult.
Next Steps
I'm pretty happy with how Jingle Smash turned out. Lots of people played it at the Mozilla All-hands and said they had fun. I did some performance optimization and was able to get the game up to about 50fps, but there is still more work to do (which I'll cover soon in another post).
Jingle Smash proved that we can make fun games that run in WebVR, and that load very quickly (on a good connection the entire game should load in
Fearless Security
Last year, Mozilla shipped Quantum CSS in Firefox, which was the culmination of 8 years of investment in Rust, a memory-safe systems programming language, and over a year of rewriting a major browser component in Rust. Until now, all major browser engines have been written in C++, mostly for performance reasons. However, with great performance comes great (memory) responsibility: C++ programmers have to manually manage memory, which opens a Pandora’s box of vulnerabilities. Rust not only prevents these kinds of errors, but the techniques it uses to do so also prevent data races, allowing programmers to reason more effectively about parallel code.
In the coming weeks, this three-part series will examine memory safety and thread safety, and close with a case study of the potential security benefits gained from rewriting Firefox’s CSS engine in Rust.
What Is Memory Safety
When we talk about building secure applications, we often focus on memory safety. Informally, this means that in all possible executions of a program, there is no access to invalid memory. Violations include:
- use after free
- null pointer dereference
- using uninitialized memory
- double free
- buffer overflow
For a more formal definition, see Michael Hicks’ What is memory safety post and The Meaning of Memory Safety, a paper that formalizes memory safety.
Memory violations like these can cause programs to crash unexpectedly and can be exploited to alter intended behavior. Potential consequences of a memory-related bug include information leakage, arbitrary code execution, and remote code execution.
Managing Memory
Memory management is crucial to both the performance and the security of applications. This section will discuss the basic memory model. One key concept is pointers. A pointer is a variable that stores a memory address. If we visit that memory address, there will be some data there, so we say that the pointer is a reference to (or points to) that data. Just like a home address shows people where to find you, a memory address shows a program where to find data.
Everything in a program is located at a particular memory address, including code instructions. Pointer misuse can cause serious security vulnerabilities, including information leakage and arbitrary code execution.
Allocation/free
When we create a variable, the program needs to allocate enough space in memory to store the data for that variable. Since the memory owned by each process is finite, we also need some way of reclaiming resources (or freeing them). When memory is freed, it becomes available to store new data, but the old data can still exist until it is overwritten.
Buffers
A buffer is a contiguous area of memory that stores multiple instances of the same data type. For example, the phrase “My cat is Batman” would be stored in a 16-byte buffer. Buffers are defined by a starting memory address and a length; because the data stored in memory next to a buffer could be unrelated, it’s important to ensure we don’t read or write past the buffer boundaries.
Control Flow
Programs are composed of subroutines, which are executed in a particular order. At the end of a subroutine, the computer jumps to a stored pointer (called the return address) to the next part of code that should be executed. When we jump to the return address, one of three things happens:
- The process continues as expected (the return address was not corrupted).
- The process crashes (the return address was altered to point at non-executable memory).
- The process continues, but not as expected (the return address was altered and control flow changed).
How languages achieve memory safety
We often think of programming languages on a spectrum. On one end, languages like C/C++ are efficient, but require manual memory management; on the other, interpreted languages use automatic memory management (like reference counting or garbage collection [GC]), but pay the price in performance. Even languages with highly optimized garbage
The Open Web Platform is a careful and fragile construction billions of people, including millions of implementors rely on. HTML, CSS, JavaScript, the Document Object Model, the Web API and more are all standardized one way or another; that means vendors and stakeholders gather around a table to discuss all changes and that these changes must pass quality and/or availability criteria to be considered "shippable".
One notable absent from the list of Web Standards is WebExtensions. WebExtensions are the generalized name of Google Chrome Extensions that became mainstream when Google achieved dominance over the desktop browser market and when Mozilla abandoned its own, and much more powerful, addons system based on XUL and privileged scripts.
As a reminder, the WebExtension API allows coders to implement extensions to the browser based on:
- HTML/CSS/JS for each and every dialog created by the extension, including the ones "integrated" into the browser's UI
- a dual model with "background scripts" with more privileges than "content scripts" that get added to visited web pages
- a new API (the WebExtension API) that offers - and rather strictly controls - access to information that is not otherwise reachable from JavaScript
- a permissions model that declare what part of the aforementioned API the extension uses and which remote URLs the embedded scripts can access
- a URL model that puts everything in the extension under a
chrome-extension://URL - a review process (on the Google Chrome Extension store) supposed to block harmful codes and more
A while ago, at a time Microsoft still had its own rendering engine, it initiated a Community Group on WebExtensions at the World Wide Web Consortium (W3C). With members from most browser vendors plus a few others, this seemed to be a very positive move not only for implementors but also for users.
But unfortunately, that effort went nowhere. Lack of commitment from other browser vendors and in particular Google, Microsoft abandoning its own rendering engine, lax Community Group instead of a formal W3C Working Group, the WebExtension draft specification has been in limbos for a while now and WebExtensions clearly remain the poor parent of Web Standards even if most people have at least one browser extension installed (usually some sort of ad-blocker).
Today, Google is impulsing a deep change in its WebExtension model:
- Background HTML pages will be deprecated in favor of ServiceWorkers. That change alone will imply a complete rearchitecture of existing extensions and will also impact their ability to create and deal with the dialogs their UX model requires.
- The
webRequestAPI that billions of users activate on a daily basis to block advertisement, trackers or undesirable content, is at stake and should be replaced by a declarartive new API that will not allow to monitor the requested resources any more. At a time the advertisement model on the Web is harmed by ad blockers, one can only wonder if this change is triggered only by technical considerations or if ad strategy is also behind it... Furthermore, it will be limited to a few dozens of thousands of declarations, which is far below the number of trackers and advertisement scripts available in the wild today. - Some heavily used API will be removed, without consideration for usage metrics or change cost to implementors
- Even the description of the top level of an extension (aka the "browser action" and the "page action") will change and impact extension vendors
- All of that is for the time being decided on the Google side alone, with little or no visible contact with the other WebExtension host (Mozilla) or the thousands of WebExtension (free or commercial) providers. There is even a "migration plans" document but it's not publicly available, the link being access-restricted
On the webRequest part specifically, all major actors of
the ad-blocking and security landscape are screaming (see also the chromium-extensions Google group). Us at Privowny are
also deeply concerned by the v3 proposed changes. Even Amnesty
International complained in a recent message! To me, the most important message posted in reply to the proposed changes is the following one:
Hi, we are the developer of a child-protection add-on, which strives to make the Internet safer for minors. This change would cripple our efforts on Chrome.
Talk about "don't be
Yesterday, I had attracted audience enough to fill up the largest presentation room GOTO 10 has, which means about one hundred interested souls.
The subject of the day was HTTP/3. The event was filmed with a mevo camera and I captured the presentation directly from my laptop as well, and I then stitched together the two sources into this final version late last night. As you’ll notice, the sound isn’t awesome and the rest of the “production” isn’t exactly top notch either, but hey, I don’t think it matters too much.
The slide set can also be viewed on slideshare.
https://daniel.haxx.se/blog/2019/01/23/http-3-talk-on-video/
Virtual reality (VR) seems complicated, but with a few JavaScript libraries and tools, and the power of WebGL, you can make very nice VR scenes that can be viewed and shared in a headset like an Oculus Go or HTC Vive, in a desktop web browser, or on your smartphone. Let me show you how:
In this new YouTube series, How to make a virtual reality project in your browser with three.js and WebVR, I’ll take you through building an interactive birthday card in seven short tutorials, complete with code and examples to get you started. The whole series clocks in under 60 minutes. We begin by getting a basic cube on the screen, add some nice 3D models, set up lights and navigation, then finally add music.
All you need are basic JavaScript skills and an internet connection.
Here’s the whole series. Come join me:
1: Learn how to build virtual reality scenes on the web with WebVR and JavaScript
2: Set up your WebVR workflow and code to build a virtual reality birthday card
3: Using a WebVR editor (Spoke) to create a fun 3D birthday card
4: How to create realistic lighting in a virtual reality scene
5: How to move around in virtual reality using teleportation to navigate your scene
6: Adding text and text effects to your WebVR scene with a few lines of code
7: How to add finishing touches like sound and sky to your WebVR scene
To learn how to make more cool stuff with web technologies, subscribe to Mozilla Hacks on YouTube. And if you want to get more involved in learning to create mixed reality experiences for the web, you can follow @MozillaReality on twitter for news, articles, and updates.
The post How to make VR with the web, a new video series appeared first on Mozilla Hacks - the Web developer blog.
https://hacks.mozilla.org/2019/01/how-to-make-vr-with-the-web-video-series/
Since 2015, the Mozilla Foundation has incubated The Coral Project to support journalism and improve online dialog around the world through privacy-centered, open source software. Originally founded as a two-year collaboration between Mozilla, The New York Times and the Washington Post, it became entirely a Mozilla project in 2017.
Over the past 3.5 years, The Coral Project has developed two software tools, a series of guides and best practices, and grown a community of journalism technologists around the world advancing privacy and better online conversation.
Coral’s first tool, Ask, has been used by journalists in several countries, including the Spotlight team at the Boston Globe, whose series on racism used Ask on seven different occasions, and was a finalist for the Pulitzer Prize in Local Reporting.
The Coral Project’s main tool, the Talk platform, now powers the comments for nearly 50 newsrooms in 11 countries, including The Wall Street Journal, the Washington Post, The Intercept, and the Globe and Mail. The Coral Project has also collaborated with academics and technologists, running events and working with researchers to reduce online harassment and raise the quality of conversation on the decentralized web.
After 3.5 years at Mozilla, the time is right for Coral software to move further into the journalism space, and grow with the support of an organization grounded in that industry. And so, in January, the entire Coral Project team will join Vox Media, a leading media company with deep ties in online community engagement.
Under Vox Media’s stewardship, The Coral Project will receive the backing of a large company with an unrivaled collection of journalists as well as experience in the area of Software as a Service. This combination will help specifically to grow the adoption of Coral’s commenting platform Talk, while continuing as an open source project that respects user privacy.
The Coral Project has built a community of journalists and technologists who care deeply about improving the quality of online conversation. Mozilla will continue to support and highlight the work of this community as champions of a healthy, humane internet that is accessible to all.
We are excited for the new phase of The Coral Project at Vox Media, and hope you will join us in celebrating its success so far, and in supporting our shared vision for a better internet.
The post The Coral Project is Moving to Vox Media appeared first on The Mozilla Blog.
https://blog.mozilla.org/blog/2019/01/22/the-coral-project-is-moving-to-vox-media/
On the morning of 19 February, Mozilla will host the second of our Mozilla Mornings series – regular breakfast meetings where we bring together policy experts, policymakers and practitioners for insight and discussion on the latest EU digital policy developments. This session will be devoted to disinformation and online advertising.
Our expert panel will seek to unpack the relation between the two and explore policy solutions to ensure a healthy online advertising ecosystem.
Speakers
MEP Marietje Schaake, ALDE MEP
Clara Hanot, EU Disinfo Lab
Raegan MacDonald, Mozilla
Moderated by Brian Maguire, EURACTIV
Logistical information
19 February 2019
08:30-10:00
Sliversquare Europe, Square de Mee^us 35
Register your attendance here
The post Brussels Mozilla Mornings – Disinformation and online advertising: an unhealthy relationship? appeared first on Open Policy & Advocacy.
Last year I implemented a new feature for Firefox Telemetry that changes how we can collect and analyze data with different requirements in regard to user privacy & frequency. This post will shine some light on the (rather simple) implementation and usage.
Intro: What is Firefox Telemetry?
In order to understand how Firefox performs in the wild, it can collect a bunch of performance metrics and other information. How and why we do this and what data we collect is explained in more detail in a blog post by Rebecca Weiss, Director of Data Science here at Mozilla: It’s your data, we’re just living in it.
I work on the Telemetry component inside Firefox. It provides APIs that are used by the various other parts of the browser to gather data and is responsible for storing, collecting and sending this data in what we call "pings", a periodic collection of measurements. Telemetry data is only ever sent out if the user agreed to it (see "Data Collection and Use" in the "Privacy & Security" preferences of your Firefox).
Most data is collected in one of 3 different formats: histograms, scalars & events (see our collection overview). Firefox sends this data in the "main" ping once in a while (usually roughly daily) and clears out the stored data locally. A "main" ping always corresponds to a subsession, which itself is part of a session. This is further explained in our Session concepts.
People working with the data can therefore make certain assumptions on how to interpret the data from multiple pings across sessions and subsessions.
The problem
Some data should not be correlated with other data due to privacy concerns. So far we had to push Telemetry users to create custom pings and keep track of their own data. If they rely on scalars or histograms as recorded by Telemetry, but send a custom ping in different intervals, they can't make valid assumptions about the metrics, as they might have been reset in between. This leads to weird hacks or unnecessary code duplication. Additionally we can't provide any help or support for custom data and our tools can't handle it automatically (e.g. to generate dashboards).
A solution
Multi-Store.
Every metric was always tied to the schedule of the "main" ping (read: subsession/session). Our multi-store solution now enables metrics to be associated with multiple stores at once, defaulting to be in the "main" ping only. Any user with custom requirements can now select metrics to be included in their custom store (which is still subject to Data Collection Review). Telemetry is still responsible for actual data storage and the APIs, but now the custom ping is responsible for collecting, clearing and periodically sending this data.
An example
To demonstrate how this is used, let's create a custom ping, which will include one metric of its own and one that's also available in the "main" store (and thus the "main" ping).
We start by adding a new metric to Scalars.yaml:
tick_times_rand: bug_numbers: - 0 description: "A random value at every tick"
I am excited to announce that Roxi Wen is joining Mozilla Corporation as our Chief Financial Officer (CFO) next month.
As a wholly-owned subsidiary of the non-profit Mozilla Foundation, the Mozilla Corporation, with over 1,000 full-time employees worldwide, creates products, advances public policy and explores new technology that give people more control over their lives online, and shapes the future of the global internet platform for the public good.
As our CFO Roxi will become a key member of our senior executive team with responsibility for leading financial operations and strategy as we scale our mission impact with new and existing products, technology and business models to better serve our users and advance our agenda for a healthier internet.
“I’m thrilled to join Mozilla at such a pivotal moment for the technology sector,” said Roxi Wen. “With consumers demanding more and better from the companies that supply the technology they rely upon, Mozilla is well-positioned to become their go-to choice and I am eager to lend my financial know-how to this effort.”
Roxi comes to Mozilla from Elo Touch Solutions, where she was CFO for the private equity-backed (The Gores Group) $400 million global manufacturer of touch screen computing systems. She brings to Mozilla experience across varying sectors from technology to healthcare to banking having held senior-level positions at GE Energy, Medtronic and Royal Bank of Canada.
Roxi is a CFA charterholder, earned a Bachelor of Economics from Xiamen University, China, a MBA in Finance and Strategy from the Carlson School of Management at the University of Minnesota. When she assumes her role in mid-February, Roxi will be based in our Mountain View, California headquarters.
Please join me in welcoming Roxi to Mozilla.
The post Welcome Roxi Wen, our incoming Chief Financial Officer appeared first on The Mozilla Blog.
https://blog.mozilla.org/blog/2019/01/22/welcome-roxi-wen-our-incoming-chief-financial-officer/
Please meet our newest Friend of Add-ons, Shivam Singhal! Shivam became involved with the add-ons community in April 2017. Currently, he is an extension developer, Mozilla Rep, and code contributor to addons.mozilla.org (AMO). He also helps mentor good-first-bugs on AMO.
“My skill set grew while contributing to Mozilla,” Shivam says of his experiences over the last two years. “Being the part of a big community, I have learned how to work remotely with a cross-cultural team and how to mentor newbies. I have met some super awesome people like [AMO engineers] William Durand and Rebecca Mullin. The AMO team is super helpful to newcomers and works actively to help them.”
This year, he’s looking forward to submitting patches to the WebExtensions API and Add-ons Manager in Firefox, and mentoring more new code contributors. Shivam has advice for anyone who is interested in contributing to Mozilla’s add-ons projects. “If you are shy or not feeling comfortable commenting on an issue, you can fill out the add-ons contributor survey and someone will help you get started. That’s what I did. You can also check https://whatcanidoformozilla.org for other ways to get involved.”
In his free time, Shivam enjoys watching stand-up comedy and sci-fi web series, exploring food at cafes, and going through pull requests on the AMO frontend repository.
Thanks for all of your contributions, Shivam! Your enthusiasm for the add-ons ecosystem is contagious, and it’s been a pleasure watching you grow.
To learn more about how to get involved with the add-ons community, check out our Contribute wiki.
The post Friend of Add-ons: Shivam Singhal appeared first on Mozilla Add-ons Blog.
https://blog.mozilla.org/addons/2019/01/21/friend-of-add-ons-shivam-singhal/
I trust you’ve heard by now that HTTP/3 is coming. It is the next destined HTTP version, targeted to get published as an RFC in July 2019. Not very far off.
HTTP/3 will not be done over TCP. It will only be performed over QUIC, which is a transport protocol replacement for TCP that always is done encrypted. There’s no clear-text version of QUIC.
TLS 1.3
The encryption in QUIC is based on TLS 1.3 technologies which I believe everyone thinks is a good idea and generally the correct decision. We need to successively raise the bar as we move forward with protocols.
However, QUIC is not only a transport protocol that does encryption by itself while TLS is typically (and designed as) a protocol that is done on top of TCP, it was also designed by a team of engineers who came up with a design that requires APIs from the TLS layer that the traditional TLS over TCP use case doesn’t need!
New TLS APIs
A QUIC implementation needs to extract traffic secrets from the TLS connection and it needs to be able to read/write TLS messages directly – not using the TLS record layer. TLS records are what’s used when we send TLS over TCP. (This was discussed and decided back around the time for the QUIC interim in Kista.)
These operations need APIs that still are missing in for example the very popular OpenSSL library, but also in other commonly used ones like GnuTLS and libressl. And of course schannel and Secure Transport.
Libraries known to already have done the job and expose the necessary mechanisms include BoringSSL, NSS, quicly, PicoTLS and Minq. All of those are incidentally TLS libraries with a more limited number of application users and less mainstream. They’re also more or less developed by people who are also actively engaged in the QUIC protocol development.
The QUIC libraries in progress now are typically using either one of the TLS libraries that already are adapted or do what ngtcp2 does: it hosts a custom-patched version of OpenSSL that brings the needed functionality.
Matt Caswell of the OpenSSL development team acknowledged this situation already back in September 2017, but so far we haven’t seen this result in updated code shipped in a released version.
curl and QUIC
curl is TLS library agnostic and can get built with around 12 different TLS libraries – one or many actually, as you can build it to allow users to select TLS backend in run-time!
OpenSSL is without competition the most popular choice to build curl with outside of the proprietary operating systems like macOS and Windows 10. But even the vendor-build and provided mac and Windows versions are also built with libraries that lack APIs for this.
With our current keen interest in QUIC and HTTP/3 support for curl, we’re about to run into an interesting TLS situation. How exactly is someone going to build curl to simultaneously support both traditional TLS based protocols as well as QUIC going forward?
I don’t have a good answer to this yet. Right now (assuming we would have the code ready in our end, which we don’t), we can’t ship QUIC or HTTP/3 support enabled for curl built to use the most popular TLS libraries! Hopefully by the time we get our code in order, the situation has improved somewhat.
This will slow down QUIC deployment
I’m personally convinced that this little API problem will be friction enough when going forward that it will slow down and hinder QUIC deployment at least initially.
When the HTTP/2 spec shipped in May 2015, it introduced a dependency on the fairly new TLS extension called ALPN that for a long time caused head aches for server admins since ALPN wasn’t supported in the OpenSSL versions that was typically installed and used at the time, but you had to upgrade OpenSSL to version 1.0.2 to get that
We have recently implemented a solution to integrate code coverage results into Phabricator.
Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions.
For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using:
$ mach try fuzzy --full
and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option.
Here is an example of a try build which produced the coverage information for my revision:
Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on all commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well.
The coverage information is shown on Phabricator both at a high-level view, in the Revision Contents section, and at a detailed view in the Diff section.
The Revision Contents section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. Here’s the screenshot of the section from my revision:
The Diff section instead shows the coverage line per line, coloring the bar on the right-hand side. Orange corresponds to uncovered lines, light blue corresponds to non-executable lines (e.g. a comment, a definition, and so on), dark blue corresponds to covered lines. When hovering the bar, the corresponding line is highlighted in the same color. The following screenshots show excerpts of my revision, with covered, uncovered and non-executable lines:
https://marco-c.github.io/2019/01/21/code-coverage-phabricator.html
We have recently implemented a solution to integrate code coverage results into Phabricator.
Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions.
For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using:
$ mach try fuzzy --full
and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option.
Here is an example of a try build which produced the coverage information for my revision:
[ïîêàçàòü]
Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on all commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well.
The coverage information is shown on Phabricator both at a high-level view, in the Revision Contents section, and at a detailed view in the Diff section.
The Revision Contents section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. Here’s the screenshot of the section from my revision:
[ïîêàçàòü]
The Diff section instead shows the coverage line per line, coloring the bar on the right-hand side. Orange corresponds to uncovered lines, light blue corresponds to non-executable lines (e.g. a comment, a definition, and so on), dark blue corresponds to covered lines. When hovering the bar, the corresponding line is highlighted in the same color. The following screenshots show excerpts of my revision, with covered, uncovered and non-executable lines:
[ïîêàçàòü]
[ïîêàçàòü]
https://release.mozilla.org/tooling/codecoverage/2019/01/20/code-coverage-on-phabricator.html
I had a very busy 2018. So busy, in fact, that I have not been able to devote any time to actually discussing what I worked on! I had intended to write these posts during the end of December, but a hardware failure delayed that until the new year. Alas, here we are in 2019, and I am going to do a series of retrospectives on last year’s work, broken up by quarter.
(Links to future posts will go here)
Overview
The general theme of my work in 2018 was dealing with the DLL injection problem: On Windows, third parties love to forcibly load their DLLs into other processes – web browsers in particular, thus making Firefox a primary target.
Many of these libraries tend to alter Firefox processes in ways that hurt the stability and/or performance of our code; many chemspill releases have gone out over the years to deal with these problems. While I could rant for hours over this, the fact is that DLL injection is rampant in the ecosystem of Windows desktop applications and is not going to disappear any time soon. In the meantime, we need to be able to deal with it.
Some astute readers might be ready to send me an email or post a comment about how ignorant I am about the new(-ish) process mitigation policies that are available in newer versions of Windows. While those features are definitely useful, they are not panaceas:
- We cannot turn on the “Extension Point Disable” policy for users of assistive technologies; screen
readers rely heavily on DLL injection using
SetWindowsHookExandSetWinEventHook, both of which are covered by this policy; - We could enable the “Microsoft Binary Signature” policy, however that requires us to load our own DLLs first before enabling; once that happens, it is often already too late: other DLLs have already injected themselves by the time we are able to activate this policy. (Note that this could easily be solved if this policy were augmented to also permit loading of any DLL signed by the same organization as that of the process’s executable binary, but Microsoft seems to be unwilling to do this.)
- The above mitigations are not universally available. They do not help us on Windows 7.
For me, Q1 2018 was all about gathering better data about injected DLLs.
Learning More About DLLs Injected into Firefox
One of our major pain points over the years of dealing with injected DLLs has been that the vendor of the DLL is not always apparent to us. In general, our crash reports and telemetry pings only include the leaf name of the various DLLs on a user’s system. This is intentional on our part: we want to preserve user privacy. On the other hand, this severely limits our ability to determine which party is responsible for a particular DLL.
One avenue for obtaining this information is to look at any digital signature that is embedded in the DLL. By examining the certificate that was used to sign the binary, we can extract the organization of the cert’s owner and include that with our crash reports and telemetry.
In bug 1430857 I wrote a bunch of code that enables us to extract that information from signed binaries using the Windows Authenticode APIs. Originally, in that bug, all of that signature extraction work happened from within the browser itself, while it was running: It would gather the cert information on a background thread while the browser was running, and include those annotations in a subsequent crash dump, should such a thing occur.
After some reflection, I realized that I was not gathering annotations in the right place. As an example, what if an injected DLL were to trigger a crash before the background thread had a chance to grab that DLL’s cert information?
I realized that the best place to gather this information was in a post-processing step after the
crash dump had been generated, and in fact we already had the right mechanism for doing so: the
minidump-analyzer program was already doing post-processing on Firefox crash dumps before sending
them back to Mozilla. I moved the signature extraction and crash annotation code out of Gecko and
into the analyzer in bug 1436845.
(As an aside, while working on the minidump-analyzer, I found some problems with how it handled
command line arguments: it was assuming that main passes its argv as UTF-8, which is not true on
Windows. I fixed those issues in bug
Support for a long awaited GNU C extension, asm goto, is in the midst of landing in Clang and LLVM. We want to make sure that we release a high quality implementation, so it’s important to test the new patches on real code and not just small test cases. When we hit compiler bugs in large source files, it can be tricky to find exactly what part of potentially large translation units are problematic. In this post, we’ll take a look at using C-Reduce, a multithreaded code bisection utility for C/C++, to help narrow done a reproducer for a real compiler bug (potentially; in a patch that was posted, and will be fixed before it can ship in production) from a real code base (the Linux kernel). It’s mostly a post to myself in the future, so that I can remind myself how to run C-reduce on the Linux kernel again, since this is now the third real compiler bug it’s helped me track down.
So the bug I’m focusing on when trying to compile the Linux kernel with Clang is a linkage error, all the way at the end of the build.
1
| |
Hmm…looks like the object file (drivers/spi/spidev.o), has a
section (__jump_table), that references a non-existent
symbol (.Ltmp), which looks like a temporary label that should have been
cleaned up by the compiler. Maybe it was accidentally left behind by an
optimization pass?
To run C-reduce, we need a shell script that returns 0 when it should keep
reducing, and an input file. For an input file, it’s just way simpler to
preprocess it; this helps cut down on the compiler flags that typically
requires paths (-I, -L).
Preprocess
First, let’s preprocess the source. For the kernel, if the file compiles
correctly, the kernel’s KBuild build process will create a file named in the
form path/to/.file.o.cmd, in our case drivers/spi/.spidev.o.cmd. (If the file
doesn’t compile, then
I’ve had success
hooking make path/to/file.o with
bear
then getting the compile_commands.json for the file.) I find it easiest to
copy this file to a new shell script, then strip out everything but the first
line. I then replace the -c -o with -E. chmod +x that new
shell script, then run it (outputting to stdout) to eyeball that it looks
preprocessed, then redirect the output to a .i file. Now that we have our
preprocessed input, let’s create the C-reduce shell script.
Reproducer
I find it helpful to have a shell script in the form:
- remove previous object files
- rebuild object files
- disassemble object files and pipe to grep
For you, it might be some different steps.
As the docs show,
you just need the shell script to return 0 when it should keep reducing. From
our previous shell script that pre-processed the source and dumped a .i file,
let’s change it back to stop before linking rather that preprocessing
(s/-E/-c/), and change the input to our new .i file. Finally, let’s add
the test for what we want. Since I want C-Reduce to keep reducing until the
disassmbled object file no longer references anything Ltmp related, I write:
1
| |
Now I can run the reproducer to check that it at least returns 0, which C-Reduce needs to get started:
1 2 3 | |
Running C-Reduce
Now that we have a reproducer script and input file, let’s run
Today is my last day as an employee of Mozilla. It's been almost exactly seven years - two years working on graphics and layout for Firefox, and five years working on Rust. Mostly remote, with a few stints in the Auckland office. It has been an amazing time: I've learnt an incredible amount, worked on some incredible projects, and got to work with some absolutely incredible people. BUT, it is time for me to learn some new things, and work on some new things with some new people.
Nearly everyone I've had contact with at Mozilla has been kind and smart and fun to work with. I would have liked to give thanks and a shout-out to a long list of people I've learned from or had fun with, but the list would be too long and still incomplete.
I'm going to be mostly stepping back from the Rust project too. I'm pretty sad about that (although I hope it will be worth it) - it's an extremely exciting, impactful project. As a PL researcher turned systems programmer, it really has been a dream project to work on. The Rust team at Mozilla and the Rust community in general are good people, and I'll miss working with you all terribly.
Concretely, I plan to continue to co-lead the Cargo and IDEs and Editors teams. I'll stay involved with the Rustfmt and Rustup working groups for a little while. I'll be leaving the other teams I'm involved with, including the core team (although I'll stick around in a reduced capacity for a few months). I won't be involved with code and review for Rust projects day-to-day. But I'll still be around on Discord and GitHub if needed for mentoring or occasional review; I will probably take much longer to respond.
None of the projects I've worked on are going to be left unmaintained, I'm very confident in the people working on them, on the teams I'm leaving behind, and in the Rust community in general (did I say you were awesome already?).
I'm very excited about my next steps (which I'll leave for another post), but for now I'm feeling pretty emotional about moving on from the Rust project and the Rust team at Mozilla. It's been a big part of my life for five years and I'm going to miss y'all. <3
P.S., it turns out that Steve is also leaving Mozilla - this is just a coincidence and there is no conspiracy or shared motive. We have different reasons for leaving, and neither of us knew the other was leaving until after we'd put in our notice. As far as I know, there is no bad blood between either of us and the Rust team.
http://www.ncameron.org/blog/leaving-mozilla-and-most-of-the-rust-project/
Bugzilla is a noisy data source: bugs are used to track anything, from Create a LDAP account for contributor X to Printing page Y doesn’t work. This makes it hard to know which bugs are bugs and which bugs are not bugs but e.g. feature requests, or meta bugs, or refactorings, and so on. To ease reading the next paragraphs, I’ll refer as bugbug to bugs that are actually bugs, as fakebug to bugs that are not actually bugs, and as bug to all Bugzilla bugs (bugbug + fakebug).
Why do we need to tell if a bug is actually a bug? There are several reasons, the main two being:
- Quality metrics: to analyze the quality of a project, to measure the churn of a given release, it can be useful to know, for example, how many bugbugs are filed in a given release cycle. If we don’t know which bugs are bugbugs and which are feature requests, we can’t precisely measure how many problems are found (= bugbugs filed) in a given component for a given release, we can only know the overall number, confusing bugbugs and feature work;
- Bug prediction: given the development history of the project, one can try to predict, with some measure of accuracy, which changes are risky and more likely to lead to regressions in the future. In order to do that, of course, you need to know which changes introduced problems in the past. If you can’t identify problems (i.e. bugbugs), then you can’t identify changes that introduced them!
On BMO, we have some optional keywords to identify regressions vs features, but they are not used consistently (and, being optional, they can’t be. We can work on improving the practices, but we can’t reach perfection when there is human involvement). So, we need another way to identify them. A possibility is to use handwritten rules (‘mozregression’ in comment -> regression; ‘support’ in title -> feature), which can be precise up to a certain accuracy level, but any improvement over that requires hard manual labor. Another option is to use machine learning techniques, leaving the hard work of extracting information from bug features to the machines!
The bugbug project is trying to do just that, at first with a very simple ML architecture.
We have a set of 1913 bugs, manually labelled between the two possible classes (bugbug vs nobug). We augment this manually labelled set with Bugzilla bugs containing the keywords ‘regression’ or ‘feature’, which are basically labelled already. The augmented data set contains 10818 bugs. Unfortunately we can’t use all of them indistinctly, as the dataset is unbalanced towards bugbugs, which would skew the results of the classifier, so we simply perform random under-sampling to reduce the number of bugbug examples. In the end, we have 1928 bugs.
We split the dataset into a training set of 1735 bugs and a test set of 193 bugs (90% - 10%).
We extract features both from bug fields (such as keywords, number of attachments, presence of a crash signature, and so on), bug title and comments.
To extract features from text (title and comments), we use a simple BoW model with 1-grams, using TF-IDF to lower the importance of very common words in the corpus and stop word removal mainly to speed up the training phase (stop word removal should not be needed for accuracy in our case since we are using a gradient boosting model, but it can speed up the training phase and it eases experimenting with other models which would really need it).
We are then training a gradient boosting model (these models usually work quite well for shallow features) on top of the extracted features.
This very simple approach, in a handful of lines of code, achieves ~93% accuracy. There’s a lot of room for improvement in the algorithm (it was, after all, written in a few hours…), so I’m confident we can get even better results.
This is just the first step: in the near future we are going to implement improvements in Bugzilla directly and in linked tooling so that we can stop guessing and have very accurate data.
Since the inception of bugbug, we have also added additional experimental models for other related problems (e.g. detecting if a bug is a good candidate for tracking, or predicting the component of a bug), turning bugbug into a platform for quickly building and experimenting with
Hi everyone! This week’s highlight is Glenn’s picture caching work which almost landed about a week ago and landed again a few hours ago. Fingers crossed! If you don’t know what picture caching means and are interested, you can read about it in the introduction of this newsletter’s season 01 episode 28.
On a more general note, the team continues focusing on the remaining list of blocker bugs which grows and shrinks depending on when you look, but the overall trend is looking good.
Without further ado:
Notable WebRender and Gecko changes
- Bobby fixed unbounded interner growth.
- Bobby overhauled the memory reporter.
- Bobby added a primitive highlighting debug tool.
- Bobby reduced code duplication around interners.
- Matt and Jeff continued investigating telemetry data.
- Jeff removed the minimum blob image size, yielding nice improvements on some talos benchmarks (18% raptor-motionmark-animometer-firefox linux64-qr opt and 7% raptor-motionmark-animometer-firefox windows10-64-qr opt).
- kvark fixed a crash.
- kvark reduced the number of vector allocations.
- kvark improved the chasing debugging tool.
- kvark fixed two issues with reference frame and scrolling.
- Andrew fixed an issue with SVGs that embed raster images not rendering correctly.
- Andrew fixed a mismatch between the size used during decoding images and the one we pass to WebRender.
- Andrew fixed a crash caused by an interaction between blob images and shared surfaces.
- Andrew avoided scene building caused by partially decoded images when possible.
- Emilio made the build system take care of generating the ffi bindings automatically.
- Emilio fixed some clipping issues.
- Glenn optimized how picture caching handle world clips.
- Glenn fixed picture caching tiles being discarded incorrectly.
- Glenn split primitive preparation into a separate culling pass.
- Glenn fixed some invalidation issues.
- Glenn improved display list correlation.
- Glenn re-landed picture caching.
- Doug improved the way we deal with document splitting to allow more than two documents.
Ongoing work
The team keeps going through the remaining blockers (14 P2 bugs and 29 P3 bugs at the time of writing).
Enabling WebRender in Firefox Nightly
In about:config, set the pref “gfx.webrender.all” to true and restart the browser.
Reporting bugs
The best place to report bugs related to WebRender in Firefox is the Graphics :: WebRender component in bugzilla.
Note that it is possible to log in with a github account.
https://mozillagfx.wordpress.com/2019/01/17/webrender-newsletter-36/
Welcome!
New localizers
- Nart Tlisha (Locales: ab, ady, uby)
- Naail, Iffam, Aiesh, Shahu, Ibrahim and Sofwath (Locale: dv)
- Kristian (Locale: vot)
Are you a locale leader and want us to include new members in our upcoming reports? Contact us!
New community/locales added
New content and projects
What’s new or coming up in Firefox desktop
The localization cycle for Firefox 66 in Nightly is approaching its end, and Tuesday (Jan 15) was the last day to get changes into Firefox 65 before it moves to release (Jan 29). These are the key dates for the next cycle:
- January 28: Nightly will be bumped to version 67.
- February 26: deadline to ship updates to Beta (Firefox 66).
As of January, localization of the Pocket add-on has moved back into the Firefox main project. That’s a positive change for localization, since it gives us a clearer schedule for updates, while before they were complex and sparse. All existing translations from the stand-alone process were imported into Mercurial repositories (and Pontoon).
In terms of prioritization, there are a couple of features to keep an eye on:
- Profile per installation: with Firefox 67, Firefox will begin using a dedicated profile for each Firefox version (including Nightly, Beta, Developer Edition, and ESR). This will make Firefox more stable when switching between versions on the same computer and will also allow you to run different Firefox installations at the same time. This introduces a set of dialogs and web pages to warn the user about the change, and explain how to sync data between profiles. Unlike other features, this targets all versions, but Nightly users in particular, since they are more likely to have multiple profiles according to Telemetry data. That’s a good reason to prioritize these strings.
- Security error pages: nothing is more frustrating than being unable to reach a website because of certificate issues. There are a lot of experiments happening around these pages and the associated user experience, both in Beta and Release, so it’s important to prioritize translations for these strings (they’re typically in netError.dtd).
What’s new or coming up in Test Pilot
As explained in this blog post, Test Pilot is reaching its end of life. The website localization has been updated in Pontoon to include messages around this change, while other experiments (Send, Monitor) will continue to exist as stand-alone projects. Screenshots is also going to see changes in the upcoming days, mostly on the server side of the project.
What’s new or coming up in mobile
Just like for Firefox desktop, the last day to get in localizations for Fennec 65 was Tuesday, Jan 15. Please see the desktop section above for more details.
Firefox iOS v15 localization deadline was Friday, January 11. The app should be released to everyone by Jan 29th, after a phased roll-out. This time around we’ve added seven new locales: Angika, Burmese,