Creating a Marker class for StreetView panoramas

TL;DR — Get it here: source, demo, documentation.

My latest addition to the google-maps-api-addon library is the PanoMarker, a marker which is able to remain at a fixed position inside of a custom StreetView panorama. It can be used to annotate points of interest (POI) inside a particular panorama regardless of the user's viewing direction. The difficulty in creating a marker that remains at a fixed position lies in the projection from a spherical panorama to a two-dimensional viewport. POIs are adressed in terms of heading and pitch angles with respect to the panorama's center. The viewport uses good old pixels for positioning elements. In this article I will elaborate on how to find the pixel coordinates on the viewport given heading and pitch angles of a POI.

StreetView uses an equirectangular projection for its panoramas. Here, latitudes and longitudes are evenly spread out along the image's vertical and horizontal axis, respectively. Note that along the "poles" (the image's top and bottom borders) all pixels are at exactly the same point in the projected sphere. The Google Maps API offers methods to obtain the user's point-of-view (POV) as well as the zoom factor which can be converted to a field-of-view (FOV) angle. By default, the view starts with a 90° FOV both horizontally and vertically (zoom level 1). With that, a fairly straightforward way of projecting angles to pixels comes to mind.

Linear Interpolation

We know that the viewport has a width of \(w\) pixels and our FOV is 90°, thus we have \(\frac{90°}{w}\) degrees per pixel in the horizontal axis (analog for the vertical one). Furthermore, we know that the viewport center corresponds to the POV angles (0°, 0°). With that knowledge we can calculate the difference to the POI coordinates and convert it into a pixel offset from the image center – done! Unfortunately this is too bad of an approximation in practice, resulting in a somewhat wobbly movement of the marker around the desired target (in this example the (0, 0) grid point):

We can observe that the marker is aligned perfectly if it's precisely in the center as well as the border regions. In between, the marker is drifting quite a bit from its correct position.

Circular Projection

Figure 28
Figure 1: 2-dimensional example of the projection

We know that we are trying to project a sphere onto a planar surface. Let's make use of that fact and do a more precise calculation. The figure on the right shows a simplified version of the problem, ignoring the pitch angle for now. Our POI has a heading of 20°. Consider the right triangle from camera (the user's POV) to the 0° and 20° points projected onto the image plane (the viewport). In this triangle, we know our desired heading \(\delta\) as well as the distance \(d\) from the camera to the plane, which can be calculated by using the bigger right triangle between camera, viewport center and viewport edge: $$ d = \frac{\frac{w}{2}}{\tan(\frac{FOV}{2})} $$

This allows us to calculate the desired pixel offset \(x = d \cdot \tan(\delta)\). Does this calculation work in practice? Kind of:

Note that the marker at (0°, 0°) is relatively stable if you turn the POV along the horizontal axis. If we consider the marker around (0°, 45°), though, the positioning is still quite bad.

Final Solution

The solution I have settled on for the Marker class is based on user3146587's fantastic post on StackOverflow. The solution performs a 3D-variant of the projection described above and works perfectly well for the whole range of pitch values:

The computation is a bit more involved. First of all, let us define the coordinate system in which we will operate. The camera will be positioned at the origin \((0, 0, 0)\). The panorma sphere is thus centered around the origin and has a diameter \(d\) (as above, the distance from camera to image plane). The panorama center with heading \(\alpha = 0°\) and pitch \(\delta = 0°\) is positioned at \((0, d, 0)\).

Figure 29
Figure 2: 3D coordinates of the POI

The first thing we want to do is calculate the 3D coordinates of both the viewport center and the POI in this coordinate system. So far we know the points' positions only in terms of heading and pitch angles w.r.t. to the panorama center. We will denote the POI at \((\alpha, \delta)\) with \(\mathbf{poi} = (x,y,z)\) and the viewport center at \((\alpha_0, \delta_0)\) with \(\mathbf{pov} = (x_0,y_0,z_0)\). As an example, the POI coordinates can be determined as follows: $$ \mathbf{poi} = \left( \begin{array}{c} x \\ y \\ z \end{array} \right) = \left( \begin{array}{c} d \cdot \cos(\delta) \cdot \sin(\alpha) \\ d \cdot \cos(\delta) \cdot \cos(\alpha) \\ d \cdot \sin(\delta) \end{array} \right) $$

The image plane on which we project the panorama is defined by its center at \((x_0,y_0,z_0)\) and its normal vector \((x_0,y_0,z_0)\) (i.e. in direction from camera to viewport center). Next, imagine a line going through the camera as well as our POI. To project the POI onto the image plane, we have to find the intersection point of that line with the image plane. By using the dot product between POI and POV vectors, we can determine the factor \(t\) with which the POI vector has to be scaled in order to meet the image plane (see also fig. 3): $$ t = \frac{\mathbf{pov} \cdot \mathbf{pov}}{\mathbf{poi} \cdot \mathbf{pov}} = \frac{|\mathbf{pov}|\cdot|\mathbf{pov}|}{|\mathbf{poi}|\cdot|\mathbf{pov}|\cdot\cos(\theta)} = \frac{1}{\cos(\theta)} = \frac{\text{hypotenuse}}{\text{adjacent}} = \frac{\text{target length}}{\text{original length}} $$

Having the intersection point, we can simply determine the vector from viewport center to the projected POI. However, since we're interested in 2D pixel offsets, we have to perform a basis transformation of the vector to the basis vectors of the image plane. This will give us the desired offsets \((u,v)\) in horizontal and vertical direction, which we can use directly to position our marker.

Figure 30
Figure 3: Intersection with the image plane
Figure 31
Figure 4: Calculation of basis vector v

To do this, we first have to determine the orthonormal basis of the image plane. In figure 3, the basis vectors are denoted with \(u\) and \(v\). Figure 4 shows the reasoning behind the formulas for \(v\), vector \(u\) can be determined in a similar fashion. We have: $$ \mathbf{u} = \left( \begin{array}{c} \cos \alpha_0 \\ -\sin \alpha_0 \\ 0 \end{array} \right), \quad \mathbf{v} = \left( \begin{array}{c} -\sin \delta_0 \sin \alpha_0 \\ -\sin \delta_0 \cos \alpha_0 \\ \cos \delta_0 \end{array} \right) $$

Finally, we can use the dot product to project our POI vector \(t\) onto said basis vectors in order to obtain the absolute pixel offsets \(x\) and \(y\) in the direction of these vectors: $$ x = \mathbf{t} \cdot \mathbf{u} = t_x \cdot u_x + t_y \cdot u_y + t_z \cdot u_z\\ y = \mathbf{t} \cdot \mathbf{v} = t_x \cdot v_x + t_y \cdot v_y + t_z \cdot v_z $$

Other Difficulties

During development, I stumbled upon two weirdnesses with the Google Maps API. First of all, the FOV angles for specific zoom levels are not documented properly. In the developer's guide we can find a table listing the FOVs for zoom levels 0 to 4. Except for zoom level 1, these angles are very inaccurate, though. Figure 5 contains a comparison between the documented angles and my own measurements. This required to write a small helper method that approximated the measured values more closely. Having a correct FOV is essential for performing the projection, thus it is very important to have accurate values.

Figure 32
Figure 5: Difference between documented and measured FOV

The marker extends the generic google.maps.OverlayView class in order to be able to work inside the google maps event framework. This class will call onAdd and onRemove callbacks when the marker is being added or removed to the Map, respectively. These methods can be overriden by our custom marker class in order to create or destroy the marker's DOM node and initialize its position. Normally, these methods are being called after as soon as the view's map is changed and the new map is in a ready state. While OverlayView.setMap() accepts both a regular Map and a StreetViewPanorama according to the documentation, the previously mentioned callbacks are only fired if the given object is in fact a google.maps.Map. In order to use an OverlayView in a StreetViewPanorama, I had to work around that issue by polling the panorama for readiness in our custom class and fire the callbacks appropriately. This was another issue not immediately clear from the API reference and should probably be fixed in the future.

Comments

Please go to Google+ to comment on this post.

By S H Posted at 00:14 (25.04.2014) UTC+2

nice

very nice! is there a good way to go from Lat/Lon to the required heading/pitch values?

+Brian Flood: From the Google Maps reference it's not quite clear if a specific heading (e.g. 0°) will always correspond to a specific real-world direction (e.g. North) in the official StreetViewPanoramas. If that is the case, though, you can get the approximate heading alpha using

alpha = arctan( lat_diff / lng_diff ).

Make sure to check the signs of the latitude/longitude differences to get the angle correctly (in the correct quadrant).

For the pitch you can use a constant value as a good approximation (e.g. 0° to place the marker close to the horizon).

If you want really exact values, you would have to take the earth's curvature into consideration, so it would get a bit more complicated ;)

thanks, I was able to get decent heading values but I could not figure out a good way to get pitch. great stuff though, cheers

+Martin Matysiak : Excellent work.  Will it be too much if I were to ask you whether you can add a function that can convert a screen click @ (x,y)  with additional params for current heading, pitch and window size and return a {heading:angleX,pitch:angleY}?

Also, I am requesting your permission to use this library on walkinto.in editor.   Shall we?

Hey +Boni Gopalan, sorry, it's been very busy around here lately; of course you can use the plugin as long as all the license conditions are met, that's why it's published open source after all ;)

Regarding the reverse povToPixel: right now I don't really have time for it, but the stackoverflow post mentioned in the source code comments does have a demo with such a function somewhere, maybe you could reuse that.

Thank you.  I have been on that SoF thread but was not as smart as you to compile the discussion into a reusable API.  Much kudos for that.  What I'll try to do is to put together a pixelToPoV method and contribute back.  

Hi guys,
I've used example from that post and come up with this function:

var pixelsToPOV = function(pos, params) {
    if(!$.isEmptyObject(pos)) {
      var w = params.width;
      var h = params.height;

      var x = pos.x;
      var y = pos.y;

      var x0 = w / 2;
      var y0 = h / 2;

      var dx = x - x0;
      var dy = y0 - y;

      var fov_x = params.fov * Math.PI / 180.0;
      var fov_y = 2 * Math.atan( h * Math.tan( fov_x / 2 ) / w );

      var dtheta_x = Math.atan( 2 * dx * Math.tan( fov_x / 2 ) / w );
      var dtheta_y = Math.atan( 2 * dy * Math.tan( fov_y / 2 ) / h );

      var theta_x0 = params.heading * Math.PI / 180.0;
      var theta_y0 = params.pitch * Math.PI / 180.0;

      var theta_x = theta_x0 + dtheta_x;
      var theta_y = theta_y0 + dtheta_y;

      params.heading = 180.0 * theta_x / Math.PI;
      params.pitch = 180.0 * theta_y / Math.PI;
    }

    return params;
  };

But, this function will not work properly when heading is to high or to low.
Does anyone know what I'm missing here?

By 吳宗翰 Posted at 09:34 (06.05.2015) UTC+2

It is a very useful technology. I'm doing the research about the measurement in panorama and I really need a mark to show the location user clicked. This technology solved my problem. Thanks!

Thanks for the awesome code! I have modified the code a little bit to have the icon as IMG rather than background. Hence it can be re-sized as per the size supplied.

This is inside the function PanoMarker.prototype.onAdd

  if (this.icon_)
  {
      var img = document.createElement('img');
      img.src = this.icon_;
      img.style.width = "100%";
      img.style.height = "100%";
      marker.appendChild(img);
      // marker.style.backgroundImage = 'url(' + this.icon_ + ')';
  }

Apart from this, I have noticed that the marker doesn't keep steady at exact location as the POV changes, it takes an elliptical deviation path. Has anyone faced or noticed this issue?

By 吳宗翰 Posted at 13:53 (25.06.2015) UTC+2

Yes, I also found this issue and I think the problem is that the focal thengh is not accurate enough. Because thefoxal thengh in google api only gave value on integer zoom level. However, the zoom level actually not a integer(In my PC I tried it is 0.33, 0.66, 0.99 etc.) Therefore, the exact focal lengh is hard to get in api document. In auther's code, the focal length is obtained by testing and optimization, but it still not exact value.

With newer versions of Maps API 
return 90 / Math.pow(2, zoom); works remarkably well.

By 吳宗翰 Posted at 16:13 (25.06.2015) UTC+2

I have tried the newer version and it still has some error when zoom level is higher than 4. In the example the author provided (http://marmat.github.io/google-maps-api-addons/panomarker/examples/fancy.html). When the maximum zoom level used, I can observed the marker will move if I changed the pov and heading. In my case I need a highly accurate position even in high zoom level, so I finally tried the different zoom level and find the best focal length for curve fitting.

@Boni where to apply this code? return 90 / Math.pow(2, zoom);
can you please post full code which is working? because I still see elliptical path when POV is changed, even with newer api. I just use https://maps.googleapis.com/maps/api/js