Page 1 of 2 12 LastLast
Results 1 to 10 of 17

Thread: iRay server on Google Compute VM not working

  1. #1
    Join Date
    Jul 2019
    Posts
    15

    Default iRay server on Google Compute VM not working

    Hi,
    I installed iRay Server on a Google Compute VM with a Tesla V100 GPU.
    Fired up the server and it didn't throw any errors.

    While I'm still having problems connecting to that iRay server instance, I tried running the Cornell Box test on it.
    It was rendering extremely slow, probably CPU render. Had to cancel after a few minutes.

    Any idea why this wouldn't work?

    Thanks,
    Kay

  2. #2
    Join Date
    Dec 2017
    Location
    Melbourne, Australia
    Posts
    302

    Default

    My first guess would be driver version. Can you provide the output of the nvidia-smi command on your machine.

  3. #3
    Join Date
    Jul 2019
    Posts
    15

    Default

    Hi,
    It turns out that rendering in an RDC session is causing the fallback to CPU render.
    Luckily iRay server did mention that in the log, so running the same cornell box test logged in to the client remotely did work much faster.

    I even managed to connect at least one of my DAZ instances to the iRay cloud server and queue a remote render. It was a matter of opening more than just port 9090 on the cloud server.

    But here are the 2 issues I ran into after that:
    I had installed DAZ Studio on the cloud server to make sure that iRay rendering actually works. And it did - but my test scene rendered only twice as fast on the clouds Tesla V100 GPU
    than it does on my own 1080/1060 rig. Not very impressive considering the cost of a V100.

    And secondly, uploading a job to the clouds iRay queue takes forever, negating the gain in render speed immediately. At least with Google Compute.
    I'm not sure why GC has such crappy upload speeds (other than using Dropbox for moving files to and from that server), but it makes cloud rendering virtually useless.

    None of these issues are faults of iRay Server, but I thought I give folks here an idea of what to expect going that route.
    Seems to me that local hardware is currently the only way to go with distributed rendering.

    - Kay
    Last edited by drgonzo; 29th July, 2019 at 20:17.

  4. #4
    Join Date
    Jul 2019
    Posts
    15

    Default

    I have disabled ECC on the V100 as iRay server suggested.
    Here's is part of the log. I wonder why it say that the v100 can not be used?



    [Mon, 29 Jul 2019 19:49:08] 3 296300.7780 | 1.5 IRAY rend info : iray photoreal CPU fallback is enabled.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.7780 | 1.5 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 2 287000.7831 | 1.8 IRAY rend warn : GPU 1 (Tesla V100-SXM2-16GB) with CUDA compute capability 7.0 cannot be used by iray photoreal.
    [Mon, 29 Jul 2019 19:49:08] 2 287000.7831 | 1.8 IRAY rend warn : There is no CUDA-capable GPU available to the iray photoreal renderer.
    [Mon, 29 Jul 2019 19:49:08] 3 287000.7831 | 1.8 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRAY rend info : CUDA device 0 (Tesla V100-SXM2-16GB): compute capability 7.0, 16 GiB total, 13.2642 GiB available, display attached
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRAY rend info : iray photoreal CPU fallback is enabled.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 3 302800.7188 | 1.3 IRAY rend info : CUDA device 0 (Tesla V100-SXM2-16GB): compute capability 7.0, 16 GiB total, 13.2642 GiB available, display attached
    [Mon, 29 Jul 2019 19:49:08] 3 302800.7188 | 1.3 IRAY rend info : iray photoreal CPU fallback is enabled.
    [Mon, 29 Jul 2019 19:49:08] 3 302800.7188 | 1.3 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.7780 | 1.5 IRAY rend info : CUDA device 0 (Tesla V100-SXM2-16GB): compute capability 7.0, 16 GiB total, 13.2642 GiB available, display attached
    [Mon, 29 Jul 2019 19:49:08] 3 296300.7780 | 1.5 IRAY rend info : iray photoreal CPU fallback is enabled.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.7780 | 1.5 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 2 287000.7831 | 1.8 IRAY rend warn : GPU 1 (Tesla V100-SXM2-16GB) with CUDA compute capability 7.0 cannot be used by iray photoreal.
    [Mon, 29 Jul 2019 19:49:08] 2 287000.7831 | 1.8 IRAY rend warn : There is no CUDA-capable GPU available to the iray photoreal renderer.
    [Mon, 29 Jul 2019 19:49:08] 3 287000.7831 | 1.8 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRAY rend info : CUDA device 0 (Tesla V100-SXM2-16GB): compute capability 7.0, 16 GiB total, 13.2642 GiB available, display attached
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRAY rend info : iray photoreal CPU fallback is enabled.
    [Mon, 29 Jul 2019 19:49:08] 3 296300.3713 | 1.9 IRT rend info : Resource assignment for host 0 has changed.
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | --------------------------------------------------
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | To get started with the Iray Server web interface:
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | (1) Open a web browser
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | (2) Visit: http://127.0.0.1:9090
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | (3) Enter your credentials to sign in
    [Mon, 29 Jul 2019 19:49:08] 3 IRAY_SERVER | --------------------------------------------------
    [Mon, 29 Jul 2019 19:49:47] 3 IRAY_SERVER | Successfully authenticated user "admin"

  5. #5
    Join Date
    Dec 2017
    Location
    Melbourne, Australia
    Posts
    302

    Default

    For an idea of performance you can refer to our benchmarks here. Based on what we have seen, double the performance of a machine with a 1080 and 1060 in it would be about right. The price of professional cards is not a reflection of performance but other factors like memory capacity, reliability in data-center environments (e.g., with passive cooling), support, certification etc). In any case, you can see both hardware and if you scroll down, cloud provider benchmarks on our website.

    For DAZ Studio, there have been improvements in this however I think they may only be in the private builds for now (they are listed in the public changelog but only released to those with private build access). Uploading will obviously be entirely dependent on the connection speed but we believe there were other factors which made it artificially slow in DAZ3D. Hopefully those are addressed in the new version. Of course all data is also cached so upload of similar data which shares information will be faster.

  6. #6
    Join Date
    Jul 2019
    Posts
    15

    Default

    Hi Paul,
    Thanks for your feedback.
    I would have hoped by comparing mflops of a 1080 or 1060 versus a V100 that the later would blow everything else away.
    Guess not.

    The slow upload speeds to the VM are systemic I'm afraid. An RDC file copy takes forever, and only moving files to Dropbox
    etc and then onto the VM is fast.
    That makes uploading complex scenes for rendering almost impossible. I would have liked to test render a more complex scene
    for comparison with my own rig, but after 5 minutes I canceled the queue.
    This may be a GC specific thing though, but I remember dropping Paperspace for the same reason. Super slow asset uploads.

    Unless the relevant scene assets are packaged and copied to the VM prior to rendering via cloud storage, I don't think GC and
    other similar cloud VM providers are usable for this purpose.
    I might as well get another rig with dual 1080's
    Last edited by drgonzo; 29th July, 2019 at 23:47.

  7. #7
    Join Date
    Dec 2017
    Location
    Melbourne, Australia
    Posts
    302

    Default

    Comparing mflops is not really that useful as a general metric. I'd definitely recommend our benchmarks are a more accurate guide since they are using the actual Iray workload. Different types of compute perform very differently on GPUs so specs generally don't tell the story. What does work though is extrapolating performance from our benchmark numbers for cards with the same architecture but different specs.

    So for example if you take a Pascal card we have benchmarked and want to guess the performance of one we have not, you can usually assume the CUDA core count will more or less linearly affect performance, clock speed has a similar effect. Note that for RTX this starts to get a little less cut and dry. For an explanation of why see our article RTX Performance Explained.

    On upload, obviously we can't influence cloud provider upload speed, however I don't see this issue on all providers, on Nimbix for example we don't see it as a general issue (provided the client-side connection is fast enough). On Nimbix we offer a pre-configured Iray Server application which can be seen being used with DAZ3D in this video.

    Of course, it will never compete with local rendering for startup time unless you have already run the scene at least once (in which case it will be entirely cached and not uploaded at all). However note that the caching is not a naive full scene cache, it caches the individual elements (even ignoring their names). So if you use a texture in many scenes it does not get uploaded again, even if in a different scene (assuming you actually persist the cache to disk).

  8. #8
    Join Date
    Jul 2019
    Posts
    15

    Default

    Thanks Paul, I'll check out Nimbix.

    Are the GPU benchmarks here in a single thread?

    Regards,
    Kay

  9. #9
    Join Date
    Dec 2017
    Location
    Melbourne, Australia
    Posts
    302

    Default

    Not sure what you mean by whether the benchmarks are in a single thread.

  10. #10
    Join Date
    Jul 2019
    Posts
    15

    Default

    Just wondering if there is a quick overview of the different benchmarks.

    Never mind, I thought that you were referring to benchmarks on irayplugins.com.
    Last edited by drgonzo; 30th July, 2019 at 17:39. Reason: Figured it out

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •