The problem has nothing to do with mipmapping, actually. Allow me to illustrate:

What's going on in the first panel is that bilinear filtering is causing the pixels to be blended between black and the background color. So the transparency is blending from 0% to 100%. This is all well and good, except the default color for transparent pixels in Photoshop is white, and there's no way to change that. So the pixels are also being blended from black to white. Therefore all the in-between areas are partially transparent shades of gray. As you can see, turning off mipmaps doesn't do a thing to help--it's not causing the problem in the first place.
In the second panel, turning off bilinear filtering fixes the interpolation problem. (While introducing a blockiness problem, but that's besides the point here.) Most of the pixels are either 100% solid or 100% transparent, giving no chance for the white background to show up. However, the diagonal cut is done with anti-aliasing, so some of the pixels there are partially transparent. Again, this allows the white background to show through.
In the third panel, an alpha channel has been made and the shape pasted into it, and the background is just filled with solid black. Now, the bilinear filtering is blending the transparency from 0% to 100%, and the pixels are all black (including the transparent areas), so the problem is completely eliminated.
The fourth panel shows that the black background also fixed the issue with the antialiased diagonal cut.
It's actually possible in some cases to use plain transparency (no separate alpha channel) and eliminate the white fringes. Namely, when saving .png files for the web in Photoshop, you can specify the matte color. If your texture is primarily one color, use this for the matte color and bingo, problem solved. (Except you have to export for the web as .png and can't use the texture in .psd format, which is a bit of a pain if you make changes.)
The solution is as described in the other answers: don't use transparency, use a separate alpha channel, and extend colors in the texture outward into transparent areas by one pixel (or more).
Note that there's a bug/quirk/something where sometimes the alpha channel won't show up in Unity for some reason. If this happens, you must have an actual background layer in Photoshop (labeled Background)--simply filling in a layer with solid color isn't always enough.