U4-5379 - ScheduledPublishController has some problems

Created by Sebastiaan Janssen 21 Aug 2014, 12:42:56 Updated by Houssam Edelbi 23 Sep 2016, 12:55:23

Relates to: U4-5391

Relates to: U4-5965

  1. When the user's email address is empty it never validates the auth token correctly.

I've put in a temporary hack (2666742851c8a4a9bda5f61ab4b76ccc3585cb2f) so that scheduled publishing keeps working on UaaS but that should obviously be better. I suggest selecting the first admin user that has a valid base64 encoded password, but there might be a better way? Remember the password format is different if you use the old membership provider or legacyEncoding.

2.It seems like some people get a 404 error on the ScheduledPublishing.Start method for some reason: 2014-08-22 10:35:37,020 [7] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 10] An error occurred with the scheduled publishing System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request) at System.Net.WebClient.UploadString(Uri address, String method, String data) at Umbraco.Web.Scheduling.ScheduledPublishing.Start(ApplicationContext appContext) 2014-08-22 10:36:01,876 [7] ERROR Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute - [Thread 28] Failed to format passed in token value System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation. at System.Web.Security.Cryptography.HomogenizingCryptoServiceWrapper.HomogenizeErrors(Func`2 func, Byte[] input) at System.Web.Security.FormsAuthentication.Decrypt(String encryptedTicket) at Umbraco.Core.StringExtensions.DecryptWithMachineKey(String value) at Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute.AuthorizeCore(HttpContextBase httpContext)

2 Attachments

Download log-excerpt.txt

Comments

Shannon Deminick 21 Aug 2014, 19:15:40

@sebastiaan I can't figure out why this change (the temp hack) was necessary?

There's always going to be an admin user with an id of '0', and the purpose of this token doesn't care what is actually stored as a value for the admin password, it's just an internal way to validate that scheduled publishing is secure.

Also, there's no gaurantee that the admin user will be returned since you are not getting all users only 25 of them (should use int.max)

Anyways, if you can let me know why this change is necessary, that'd be great.


Sebastiaan Janssen 22 Aug 2014, 09:15:55

Commit made by '''Sebastiaan Janssen''' on ''2014-08-22T11:15:46+02:00'' https://github.com/umbraco/Umbraco-CMS/commit/eae00873073f20c60e355ec6e95ff6259ad2652b

#U4-5379 Fixed Due in version: 7.2.0

Some users have not set an email, don't strip out empty entries


Sebastiaan Janssen 22 Aug 2014, 09:16:34

Yeah I was completely not thinking when I did this. I just realized this does absolutely nothing to fix this as.. in AuthorizeCore it still gets the user 0. The REAL problem was not the password but the fact that the email address is empty. People who upgrade from < v7.1 will have this happen as well as the email was never mandatory before. So now I've put in the real fix and that is not to strip out empty entries. Let me know if that is problematic elsewhere, I can't think of a problem.


Sebastiaan Janssen 22 Aug 2014, 10:49:05

That might not fix it completely, the problem we're seeing on Azure is a 404, which is apparently not just on UaaS:

http://our.umbraco.org/forum/getting-started/installing-umbraco/55867-Setting-up-distributedcall-for-Umbraco-load-balancing-with-Umbraco-7

Not sure why it would 404, on my machine I could only reproduce a 401 because of the aforementioned length.


Shannon Deminick 25 Aug 2014, 06:24:45

I've replied to that Our thread, the 404 is because it doesn't know what address to use to send a request to itself. If there's nothing specified in the distributed call part of the umbraco config, it will assume that the original request used to start the app is a base url that it can use to send a request to itself. With Azure, the original request is probably a proxied address and it cannot use that one to talk to itself. This has been the behavior with scheduled tasks for quite some time, so with Azure was probably never working.

Since not all installs will be using distributed call, we'll have to probably create another config entry for this sort of thing.


Sebastiaan Janssen 25 Aug 2014, 06:30:04

@shandem Okay, but we're getting that same 404 on UaaS where we've never set up distributed calls. Not able to produce on local though, strange.

2014-08-21 11:43:22,344 [25] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 66] An error occurred with the scheduled publishing System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request) at System.Net.WebClient.UploadString(Uri address, String method, String data) at Umbraco.Web.Scheduling.ScheduledPublishing.Start(ApplicationContext appContext)


Shannon Deminick 25 Aug 2014, 06:40:10

Before scheduled publishing worked without making a request, it just did it on a timer. The logic was streamlined so that all scheduled things worked the same way so the logic that was doing the scheduled tasks is now also doing the scheduled publishing. Being naive, I thought that this had always worked since the logic had been that way for quite some time, however it seems that it doesn't in some environments. Also worth noting is the keep alive service works this way too.

I've just added some logging so we know what the ApplicationContext.Current.OriginalRequestUrl is set to on app startup

We can hack this though without changing versions if you want to test on UaaS, you'd just have to use reflection to get the result of: ApplicationContext.Current.OriginalRequestUrl and see what it's set to. You could also use reflection to set that property as well.


Sebastiaan Janssen 25 Aug 2014, 06:53:22

The result is: 2014-08-25 08:49:20,628 [87] INFO Umbraco.Web.UmbracoModule - [Thread 99] Setting OriginalRequestUrl: dev-mylogtest.umbraco.io:80/umbraco

This is indeed a correct Url. I wonder if I get a 404 because this URL is password protected? Will try on unprotected as well.


Sebastiaan Janssen 25 Aug 2014, 06:59:27

Ah, but it's the 401 (which was my own fault) so it should be good now (with the new fix I put in for empty email addresses). Will run one last test with that.


Shannon Deminick 25 Aug 2014, 07:05:27

Ok great. I suppose it might still be worth having a config param somewhere to specify a custom base url in case the host is using ARR or something and the site isn't configured for LB?


Sebastiaan Janssen 25 Aug 2014, 07:13:52

So the 401 is now fixed and I couldn't actually repro the 404.. (weird, but it's Morten that got it, I never saw it, so it might have been an upgrade problem.

I get a new 401 now though:

2014-08-25 09:06:16,917 [105] ERROR Umbraco.Web.Scheduling.KeepAlive - [Thread 102] Error in ping System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request) at System.Net.WebClient.DownloadString(Uri address) at System.Net.WebClient.DownloadString(String address) at Umbraco.Web.Scheduling.KeepAlive.Start(Object sender)

Will investigate, this only appears on the password protected site so it might well be that.

Strangely I don't see the Setting OriginalRequestUrl log on the live site after publishing, yet it does appear on the dev site. Not sure why that would be different.

You think this might be due to ARR?? UaaS does use that. Definitely could be good to have a config setting for it (disclaimer: I have no idea how ARR works! ;)).


Shannon Deminick 25 Aug 2014, 07:24:40

Yeah, that 401 looks like it's because you have a password protected site.

Currently the OriginalRequestUrl is set with this:

string.Format("{0}:{1}{2}", httpContext.Request.ServerVariables["SERVER_NAME"], httpContext.Request.ServerVariables["SERVER_PORT"], IOHelper.ResolveUrl(SystemDirectories.Umbraco));

Depending on how ARR is configured and what it's actually doing, the originally requested URL that the end user types in might get re-proxied to the server and therefore these values might actually be different from what the user typed in.

I'll create another task for the config bit


Sebastiaan Janssen 25 Aug 2014, 07:33:01

Yup can repro when I just do a webrequest to the current site in an apicontroller so this has always been like this, no problem then!

AFAIK the URL on UaaS never changes. Anyway, seem like it would be good to be able to configure it anyway for other setups.


Sebastiaan Janssen 25 Aug 2014, 13:19:23

Commit made by '''Sebastiaan Janssen''' on ''2014-08-25T14:31:50+02:00'' https://github.com/umbraco/Umbraco-CMS/commit/2daacd8d57e0d90172d95697f028d95691d16d9c

#U4-5379 Fixed Due in version: 7.1.6

Some users have not set an email, don't strip out empty entries


Anders Brohäll 16 Sep 2014, 09:10:41

Im still getting the 404-entries in the log. Using 7.1.6:

2014-09-16 11:05:19,256 [40] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [Thread 22] An error occurred with the scheduled publishing System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request) at System.Net.WebClient.UploadString(Uri address, String method, String data) at Umbraco.Web.Scheduling.ScheduledPublishing.Start(ApplicationContext appContext)

... on a regular Windows Server 2008 R2 with standard IIS.


Sebastiaan Janssen 16 Sep 2014, 09:27:09

@anders Is your database user by any chance not using the en-US locale?

See: http://our.umbraco.org/forum/ourumb-dev-forum/bugs/47059-Trace-Log-filling-with-UmbracoCorePersistenceUmbracoDatabase-warning

And: http://issues.umbraco.org/issue/U4-4159


Anders Brohäll 16 Sep 2014, 09:43:54

Nah, turned out i was redirecting ^~/(.*)/schedule in urlrewriting. Sorry for that, it works as expected. Sorry for taking up your time!


Nicholas Westby 09 Apr 2015, 17:35:53

@sebastiaan FYI, I'm seeing this 404 error in an Umbraco 7.1.6 installation (see log excerpt attached). My admin user has an email set. Note however that I don't have a hostname set on my home node (just responds to any hostname), which seems like it could be related to this issue.


Nicholas Westby 09 Apr 2015, 20:24:50

I just added a hostname to the home node and recycle the app pool. Still seeing the 404 errors in the log.


Shannon Deminick 10 Apr 2015, 00:26:14

@Knickerbocker as this issue is closed and if you can replicate this issue on the latest v7 version, please open an new issue with details on how to replicate.


Martin Griffiths 28 Jul 2015, 09:36:54

I've deployed a project on 7.2.6 and i'm getting this error every minute..

2015-07-28 10:34:47,823 [411] ERROR Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute [(null)] - [T411/D5] Failed to format passed in token value System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation. at System.Web.Security.Cryptography.HomogenizingCryptoServiceWrapper.HomogenizeErrors(Func`2 func, Byte[] input) at System.Web.Security.FormsAuthentication.Decrypt(String encryptedTicket) at Umbraco.Core.StringExtensions.DecryptWithMachineKey(String value) at Umbraco.Web.Mvc.AdminTokenAuthorizeAttribute.AuthorizeCore(HttpContextBase httpContext)


Shannon Deminick 28 Jul 2015, 09:40:37

@martinjgriffiths First time I've ever heard of this happening. If you have steps to replicate please create a new issue, perhaps you have some custom machine key settings, specific environment setup, etc... Please provide details when submitting bug reports.


Sebastiaan Janssen 28 Jul 2015, 09:41:10

Also, please please upgrade to 7.2.8. :)


Martin Griffiths 28 Jul 2015, 09:55:13

I'll upgrade the project to 7.2.8 and report back


Martin Griffiths 28 Jul 2015, 10:04:50

Just prior to commencing an project update I tried a few things...

  1. I added an email address to admin (0)
  2. I did a republishAll
  3. I checked all of my config files (couldnt find anything unusual)
  4. Recycled App Pool many times.

None of these things seemed to help. This is bearing in mind my logs have been filling up since yesterday afternoon.

Then at 10:45am they've stopped! Why would this be? Is it possible i've been a target for an attack?

M.


Shannon Deminick 28 Jul 2015, 10:11:14

I suppose anything is possible, if you can replicate this issue locally then obviously it's an issue, but if it's only on this one environment then it could be that. You could add some logging if you wanted to track IP addresses and URLs being requested. Our Debug logging does log all requests but not IP address... probably could add that.


Martin Griffiths 28 Jul 2015, 10:25:06

Hi Shannon

I can confirm this has never been a problem locally during build, Only on deployment.

It's happened to me on other installs. But again settled down over time. I'm concerned our sites have been the subject of a probe for usernames/passwords as Umbraco sites are quite easy to identify.

Since we've moved our sites to v7 we now use SSL and i've used URLRewrite to force a 404 for /umbraco on the root. You can now only get to the backoffice via a subdomain of our root domains. It's still only an obscuring method as they will undoubtly be found at some point, but it may keep some hackers at bay!

M.


Martin Griffiths 28 Jul 2015, 15:09:34

Hi all

I've upgraded 3 projects to 7.2.8 and all are Elma Fudd (Vewy, Vewy Quiet!)

So it looks like a success! yay!

M.


Houssam Edelbi 23 Sep 2016, 12:48:44

Hello there,

I'm using U7.1.8 and I'm having a problem with load balancing. One of the two servers we have isn't refreshing. I'm getting the same error mentioned in this article. I upgraded to 7.2.8 as recommended in this article but now i have the content picker in the content grid stopped working. Can anyone help me here please?


Houssam Edelbi 23 Sep 2016, 12:55:23

Here what is happening each time i try to click on the content picker inside the content grid.


Priority: Normal

Type: Bug

State: Fixed

Assignee: Shannon Deminick

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions: 7.1.5

Due in version: 7.1.6

Sprint:

Story Points:

Cycle: