vbO 3.0 beta "fetch cache" broken

[ctrlbrk]

vb 3.8.5
vbo 3.0 beta

php 5.3.3 w/php-fpm, compiled from source

Problem:
"Your Opcache Operator extension is functioning, however it is unable to store data. Please check your extension configuration" during 'System Test'

I have tried memcache, apc, and xcache --- all latest versions compiled from source --- and all return the same error. Usually, the first time I click 'system test' it is fine, the second time I click it I get the error.

I've looked at memcache (via telnet), and APC and XCache via their web interfaces, in all cases there is data present using the vbO prefix (_vbo) so the 'sets' are working, and they all show 'hits' so I assume that means the 'gets' are working, yet I get this error message from vbO.

I'd like to learn more about why this is happening and solve it. I see from many other posts on vb.org that many people have the same problem. Many people do not. No one seems to be able to figure out why, well I am here to help figure it out

I think the first step is to determine if the set/get is actually working. Can dragonbyte please provide me a complete .php file to do a proper test, stand-alone from the 'system test' feature, so we can determine if it is truly broken or not. Call this a debug php file solely for this purpose.

Mike

 

[blandt]

vb 3.8.5
vbo 3.0 beta

php 5.3.3 w/php-fpm, compiled from source

Problem:
"Your Opcache Operator extension is functioning, however it is unable to store data. Please check your extension configuration" during 'System Test'

I have tried memcache, apc, and xcache --- all latest versions compiled from source --- and all return the same error. Usually, the first time I click 'system test' it is fine, the second time I click it I get the error.

I've looked at memcache (via telnet), and APC and XCache via their web interfaces, in all cases there is data present using the vbO prefix (_vbo) so the 'sets' are working, and they all show 'hits' so I assume that means the 'gets' are working, yet I get this error message from vbO.

I'd like to learn more about why this is happening and solve it. I see from many other posts on vb.org that many people have the same problem. Many people do not. No one seems to be able to figure out why, well I am here to help figure it out

I think the first step is to determine if the set/get is actually working. Can dragonbyte please provide me a complete .php file to do a proper test, stand-alone from the 'system test' feature, so we can determine if it is truly broken or not. Call this a debug php file solely for this purpose.

Mike

ctrlbrk

You need to configure php.ini to use APC, xcache etc ..

It depends which cache you want to use.

for memcache make sure you know which port it's running on and it's enabled at the VB config.php
run this at the command prompt :

ps -A | grep memcached

If it's working you will get its listing

then go to config.php scroll down to the memcached section (well commented throughout) and uncomment that section and make sure the port you're listening to correspond to what you got with the ps command (usually 11211 if you haven't changed the default.

For APC

go to php.ini and ad this

apc.enabled=1
apc.shm_segments=1
apc.shm_size=128
apc.ttl=7200
apc.user_ttl=7200
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1

This should work for most configurations (google around for what's best for yours).
to find your php.ini file (since it's a custom built fpm) go to your vb admin -- maintenance --> view php info
and find your php.ini path

Same thing for xcache ....

 

[ctrlbrk]

Hi,

lol. This is not the case of a noob that doesn't know how to configure it.

I know how to do it. As I said in my op, I've compiled all from source and verified via their admin interface individually that they are showing data. They are all running correctly, but vbO is saying otherwise. Please re-read my original post.

Mike

 

[ctrlbrk]

blandt, btw this error is a known issue for vbO team, it is all over the thread on vb.org, but no solution yet. I am here as I would like to help the dev team find a solution if possible.

Mike

 

[blandt]

hmmm weired ... I have no problems so far .... but i'm running vb4

 

[ctrlbrk]

hmmm weired ... I have no problems so far .... but i'm running vb4

Yeah, it worked fine for me before too, but that was on a different machine, I've recently switched hardware. But, as far as I know, I've done nothing special. Just a regular source install of php 5.3.3 with php-fpm, regular pecl install of memcache/apc, regular source install of xcache, all same as before. They were both Ubuntu 10.04 even. The reason I tried memcache, apc and xcache was to try to find one that 'worked', but they all report the error.... so it seems either the error itself is bogus and should be fixed (which I think can be proved by dragonbyte giving me the debug php file I asked for on post 1), or there is some other underlying issue here that should be fixed to make the caching work.

Mike

 

[Cosmic]

ctrlbrk it's not a known issue, as the only thing the test does is just that - a test. It just looks to see if the extension is working properly, it doesn't modify or change anything at all. People are getting the errors due to misconfiguration somewhere.

As you pointed out, it worked just fine on your old server, but since moving it doesn't - that suggests an issue with the new server you are on (There are a number of potential problems, IIRC FastCGI causes issues with a lot of cache programs, and i'm sure there are several other possible server configurations that would cause issues with caching systems.)

Deceptor is moving home at the moment, but he should be on tomorrow and may be able to give you a different way to test the caching just to ensure it really is the problem =)

Cosmic

 

[blandt]

PHP 5.3.3 should be shipped with a packaged fpm , and I beleive that's where your problem is coming from.
On a test VPS I get bad gateway error when VBoptimize is on (APC, or memcached) with php 5.3.3. When VBoptimize is off no problems
rolled back to 5.3.2 VB Optimize works just fine and the problem went away.

Perhaps the issue is with 5.3.3 and APC.... there are several rumors out on google ...

 

[ctrlbrk]

ctrlbrk it's not a known issue, as the only thing the test does is just that - a test. It just looks to see if the extension is working properly, it doesn't modify or change anything at all. People are getting the errors due to misconfiguration somewhere.

On the vb.org thread, there are a lot of people with same issue, that is why I said it is a known issue. On this post:
http://www.vbulletin.org/forum/showpost.php?p=2055169&postcount=194

Deceptor said:

Configuration seems fine, as I told rajubd you might need to contact XCache on their support forums to see why it isn't working, it's a fairly common problem. If they point out what was wrong I'd love to hear what it is

But the issue is not XCache, as I have duplicated it with memcache and APC in addition to XCache.

As you pointed out, it worked just fine on your old server, but since moving it doesn't - that suggests an issue with the new server you are on (There are a number of potential problems, IIRC FastCGI causes issues with a lot of cache programs, and i'm sure there are several other possible server configurations that would cause issues with caching systems.)

No doubt, the configuration has changed. But it is a good config, a clean config. It just doesn't work with vbO's 'system test' feature.

Deceptor is moving home at the moment, but he should be on tomorrow and may be able to give you a different way to test the caching just to ensure it really is the problem =)

Thank you. I strongly suspect, based on what I've been able to test myself, that the caching is indeed working fine and vbO's 'system test' is broken in some way. But a debug file would prove that one way or another. I could write this myself just haven't had time yet.

Mike

 

[ctrlbrk]

PHP 5.3.3 should be shipped with a packaged fpm

Yes I know, this is why I went with 5.3.3 on this install, to get the benefit of that instead of using the patch on 5.3.2 like I was prior.

I beleive that's where your problem is coming from. On a test VPS I get bad gateway error when VBoptimize is on (APC, or memcached) with php 5.3.3. When VBoptimize is off no problems
rolled back to 5.3.2 VB Optimize works just fine and the problem went away.

I don't have this problem (gateway error). But if APC and memcache work on your server with vbO disabled, and don't work with vbO enabled, the problem seems to be with vbO and not with php 5.3.3.

Mike

 

[ctrlbrk]

BTW, I have no problem waiting a day or two for Deceptor, I understand he is moving and it's not like my forum is offline or something. I just want get to the bottom of this, even though I think it is working correctly already. I thought it might help the vbO team as well since so many others have reported the same issue.

Mike

 

[blandt]

Bad gateway is nginx error for bad FastCGI. cannot connect to fpm basically.
I have also noticed that it doesn't happen right away, meaning after I turn on VBop. the forum runs fine ... but after few clicks, replies and goofing around for a bit nginx stops passing php properly hence the error.

I'm just passing my notes along...

 

[ctrlbrk]

That seems to be unrelated to my issue. But you should check your php-fpm log, or the nginx log if you have intercept errors on. I always have to remind myself to rtfl!

Mike

 

[blandt]

That seems to be unrelated to my issue.
Mike

Then I think you don't even know what your issue is

That's what the log is saying "bad gateway error 503".

This is to say that the problem is not limited to vboptimize. The problem is you need to update APC and memcached along too ...

Have a read here :

http://pecl.php.net/bugs/bug.php?id=18009&edit=3

or here
http://www.devcomments.com/PHP-5-3-...1-4-Potential-cache-slam-averted-at262664.htm

or here
http://www.litespeedtech.com/support/forum/showthread.php?t=4242


etc .. etc..
The problem is your settings with APC, VBopt is not going to do the correct settings for you.

You might even need to recompile to the latest APC

 

[ctrlbrk]

I know what the problem is. vbO is reporting fetch cache failure. As for why it is reporting this, that will be something the vbO team will need to help me figure out.

As for the rest of your reply, you can take APC out of the equation as I said in post #1, the problem exists with memcache and xcache as well, it is not an APC issue.

URL #1: I am not getting out of memory errors.
URL #2: I already tried turning off cache slam. I was getting cache slam errors in the log, and turning it off solved that. But, vbO still reports the fetch issue.
URL #3: As for the nmap, I already have that properly defined, I do not have that issue.

Again, APC works fine outside of vbO. The vbO problem exists with memcache and xcache, as well as APC.

Thanks for trying to help, but this is all irrelevant. You said "the problem is your settings with APC", that is for sure not the case.

Mike

 

[Deceptor]

ctrlbrk,

Sorry for the delay, as Cosmic mentioned I've just moved house and today I'm finally not surrounded by boxes (well, not too many). First I'd like to explain to you how vB Optimise "tests" the extension to find common the issues it reports:

1) Test 1: Connection
This is fairly straight forward, vB Optimise checks the available php extension is loaded by verifying specific functions exist.

2) Test 2: Store Cache
This test ensures data can be maintained within the same PHP session, I should point out at this stage that each test vB Optimise executes is in a new session (test 3 requires this scenario), so in Test 2 vB Optimise will assign some random data to your assigned cache system and then call it back to see if it matches what was stored. 99% of the time this works without issue if the extension is running (and even if it isn't configured properly). So in reality, we test store & fetch cache here, but within the same php session.

3) Test 3: Fetch Cache
This test is an extension to Test 2, because it runs in a new session it's validating that the data assigned during test 2 still exists. As you'll find here and in Lite support forums a lot of people fail this test due to misconfiguration, 99% of the time it's usually XCache. Misconfigurations can retain the cached data during php sessions but between sessions is where they fail and this test is designed to expose that problem.

4) Test 4: Flush Cache
This test makes sure vB Optimise can successfully communicate to kill the data in your cache, however if test 3 fails this test will succeed by default since no cache is being retained and it will believe the flush worked (regardless if it does/doesn't actually work).

To add to that, vB Optimise doesn't do anything special to setup these test scenarios. We use the standard functions provided by their APIs to do 4 things: connect, store, fetch, delete. We validate the results much like a stand-alone tester you've requested.

I can fulfill your request for a stand-alone tester, what extension would you like it to test - APC?

 

[ctrlbrk]

Deceptor,

I ended up reverting back to vbO 2.3.0 for the sake of sanity (see my last reply before this in a diff thread).

This problem still exists on 2.3.0.

I am back on xCache as it was the fastest in my testing. I would appreciate you making a stand-alone test script like discussed. In this test script, will you please also incorporate $_SERVER['SERVER_ADDR'] in the output so we can see which application server is serving the request(s). I have two load balancers and proxy's up front, which then round-robin to two app servers. The two app servers each have their own mysql master server, and a replicated slave server (the two db servers are master-master). It might be best to email me (email in my profile) because I don't want to reveal too much more in a public forum.

Mike

 

[Deceptor]

Hey ctrlbrk,

XCache is locally installed so cache isn't shared between the servers which I presume you're aware of, memcache can however run in pooled state sharing between clusters. Here's the script for you however

 

[ctrlbrk]

Hey ctrlbrk,

XCache is locally installed so cache isn't shared between the servers which I presume you're aware of, memcache can however run in pooled state sharing between clusters. Here's the script for you however

Hi, I wanted an opcode cache. Yes I know xcache is local only. I decided to use xcache for the datastore and vbo as well because it benchmarked better than using memcache, probably because of the round trip latency of going to another server even on an internal vlan. I received the error from this thread title however even with memcache.

I've installed your test script, thank you. I will PM you the URL as I don't want the IP's revealed publicly. You can refresh it multiple times to see the app server IP changing. In every case, the cache seems to work properly. So, now the question is why does vbo's self-test fail?

EDIT: Edit to mention my xcache var ttl is 60 seconds, this is to minimize invalidated/out of date info on the cluster. This setting overrides your 900 second TTL it seems, just FYI.

Mike

 

[Deceptor]

Well that's interesting, but I think I know the issue here. Your load balancer setup seems to keep users bouncing from 1 server to another with each request (through Round Robin), usually that load balancer method is accompanied with sticky masks so that once a user is assigned to a server they stick to it, but this doesn't seem to be the case with your setup. Additionally, the internal test through the vB Optimise ACP is done through ajax requests, each test is sent via a HTTP request, so it's very possible they're not all being sent to the same server.

Imagine this scenario (we'll ignore the check opcache test, we know it works):

1) Server 1 gets the request to set the cache
2) Server 2 gets the request to fetch the cache (and fails)

Now running the test straight after would provide a succesful test - but the final test vB Optimise executes is a flush, so whichever server gets that request has whatever data it has stored removed (including the test variable). Technically speaking this issue wouldn't arrise with Memcache, however that depends if you had it setup as a pooled memcache within the vBulletin configuration or not. This would also explain why you see cache through telnet being set.

While running XCache on 2 servers isn't an issue, I would recommend against recording stats (as this is done in cache), since the 2 servers will get different results all the time.

In terms of resolution to the vBO test script, I think I'll make a change tonight that forces it to "retry" the test if it's being executed on a different server, this way all 4 tests will execute in one place to give a valid result - I'll make the change for v3 and v2.3 so if you like you can try it out

Meanwhile if you have any questions feel free to shoot away.